System and method for managing message campaign data

ABSTRACT

A server system comprising one or more processors and memory receives, from a client device, one or more messages for use in a message campaign, where the one or more messages include a plurality of annotations indicating human-identified features in the one or more messages. The server system initiates delivery of the message campaign to a plurality of users and receives campaign-feedback data that is indicative of user interaction, during the message campaign, that corresponds the features identified by the one or more annotations. After receiving the campaign-feedback data, the server system receives, from a requestor, a request for analysis of the features during the message campaign. In response, the server system obtains a campaign report that provides information on one or more correlations between the features and user interaction with the features during the message campaign and provides the campaign report to the requestor.

RELATED APPLICATIONS

This application claims priority to U.S. Provisional Patent ApplicationSer. No. 61/888,977, filed Oct. 9, 2013, entitled “System and Method forManaging Message Campaign Data,” which application is incorporated byreference in its entirety.

FIELD OF THE INVENTION

The present invention relates generally to the field of computer datamanagement, and in particular, to a system and method for efficientlymanaging message campaign data.

BACKGROUND OF THE INVENTION

Message campaigns that provide information to a large number ofrecipients can be run using a variety of messaging means. Deliveringmessages in a message campaign electronically (e.g., via email, SMS,social networking message, or other electronic message) is a fast andefficient way to get timely information to message campaign recipients.However, it is important for both the senders and recipients of messagecampaigns that the message campaigns be as relevant to the recipients aspossible. Thus, the organizers of message campaigns frequently collectfeedback data about the performance of message campaigns that includesboth explicit feedback data (e.g., user surveys) and implicit feedbackdata (e.g., user responsiveness to the message campaign in the form ofpurchases and/or engagement of the user with a message in the messagecampaign, or lack thereof). Electronic message campaigns provide theopportunity to collect increasing amounts of feedback data. However,conventional approaches to processing feedback data from messagecampaigns fail to take satisfactory advantage of the feedback data andthus result in future message campaigns that are less useful/relevantfor recipients and are less effective/efficient for the message campaignorganizers.

SUMMARY

As such, it would be advantageous to be able to store and processfeedback data from message campaigns more effectively and efficiently sothat message campaign organizers can create message campaigns that aremore useful and relevant to the recipients of the message campaign.However, feedback data for even a single message campaign that is sentto a large number of users can include a large amount of data.Collecting feedback data for a large number of message campaigns over along period of time can be very useful for comparing the performance ofdifferent message campaigns, however storing this data locally wouldrequire increasing amounts of storage as additional feedback data isreceived. Pre-aggregating the feedback data can reduce the rate at whichthe amount of feedback data increases and decrease the overall amount ofstorage space occupied by the feedback data. However, pre-aggregatedfeedback data has limited flexibility and can only be used forresponding to a limited set of queries (e.g., one or more predefinedqueries that do not require information that was lost during theaggregation of the data). Thus, it would be advantageous to be able tostore feedback data that can be analyzed using arbitrary queries withoutusing up local storage space. By way of example, such arbitrary queriesinclude queries that were conceived of or refined after one or moremessage campaigns have been initiated. Transmitting the feedback data toa remote storage system without pre-aggregating the data (e.g., storing“raw” feedback data) enables arbitrary queries to be run against thedata without using up local storage space. The systems and methodsdescribed herein provide an improved approach to storing and analyzingfeedback data from message campaigns that enable message campaignorganizers to create message campaigns that are more useful and relevantto the recipients and are more effective.

In accordance with some embodiments, a method is performed at a serversystem with one or more processors and memory. The method includesobtaining user profiles for a plurality of users for which messageinteraction data is stored and receiving profile-update informationindicative of a change to a respective profile of a respective user ofthe plurality of users. The method also includes, in response toreceiving the profile-update information, updating the respectiveprofile of the respective user in accordance with the profile-updateinformation, and initiating storage of profile-archive information thatenables the server system to obtain a prior version of the respectiveprofile. The method further includes, after updating the respectiveprofile of the respective user, receiving, from a requestor, a requestfor analysis of the message interaction data, and in response toreceiving the request for analysis of the message interaction dataobtaining a message-interaction report that correlates the messageinteraction data with a set of one or more versions of profiles of oneor more users that includes the prior version of the respective profileof the respective user and providing the message-interaction report tothe requestor.

In accordance with some embodiments, a method is performed at a serversystem with one or more processors and memory. The method includesinitiating delivery of a plurality of message campaigns to a pluralityof users, where a respective message campaign of the plurality ofmessage campaigns includes one or more messages addressed to at least asubset of the plurality of users. The method also includes receiving,from a plurality of client devices associated with the plurality ofusers, campaign-feedback data, where the campaign-feedback data isindicative of user interaction with the messages transmitted to theplurality of client devices as part of the plurality of messagecampaigns and after receiving the campaign-feedback data, conditionallytransmitting, when a transmission criterion is satisfied, to one or moreremote storage systems, campaign-tracking data that is based on thecampaign-feedback data.

In accordance with some embodiments, a method is performed at a serversystem with one or more processors and memory. The method includesinitiating delivery of a plurality of message campaigns to a pluralityof users, where a respective message campaign of the plurality ofmessage campaigns includes one or more messages addressed to at least asubset of the plurality of users, and receiving, from a plurality ofclient devices associated with the plurality of users, campaign-feedbackdata. The campaign-feedback data is indicative of user interaction withthe messages transmitted to the plurality of client devices as part ofthe plurality of message campaigns. The campaign-feedback data includesinformation that enables generation of campaign reports that providequantitative information about one or more of the plurality of messagecampaigns. The method also includes, after receiving thecampaign-feedback data, transmitting, to one or more remote storagesystems, campaign-tracking data that is based on the campaign-feedbackdata and after transmitting the campaign-tracking data to the one ormore remote storage systems, receiving, from a requestor, a firstrequest for analysis of the campaign-feedback information. The methodfurther includes, in response to receiving the first request foranalysis of the campaign-feedback information obtaining a campaignreport generated using the campaign-tracking data from the one or moreremote storage systems and providing the campaign report to therequestor.

In accordance with some embodiments, a method is performed at a serversystem with one or more processors and memory. The method includesinitiating delivery of a plurality of message campaigns to a pluralityof users that include a first campaign and a second campaign, where thefirst campaign and the second campaign each include one or more messagesaddressed to at least a subset of the plurality of users. The methodalso includes receiving, from a plurality of client devices associatedwith the plurality of users, campaign-feedback data for the firstcampaign and the second campaign. The campaign-feedback data isindicative of user interaction with the messages transmitted to theplurality of client devices as part of the plurality of messagecampaign. The method further includes, after receiving thecampaign-feedback data, directing storage of campaign-tracking data thatcorresponds to the campaign-feedback data and after directing storage ofthe campaign-tracking data, receiving, from a requestor, a request for acampaign report for a respective user of the plurality of users. Themethod also includes, in response to receiving the request for acampaign report for the respective user obtaining a respective campaignreport for the respective user, where the respective campaign reportincludes information indicative of the respective user's interactionswith a first campaign and the respective user's interactions with thesecond campaign and providing the respective campaign report to therequestor.

In accordance with some embodiments, a method is performed at a serversystem with one or more processors and memory. The method includesreceiving, from a client device, one or more messages for use in amessage campaign that includes sending the one or more messages to aplurality of users, where the one or more messages include a pluralityof annotations indicating human-identified features in the one or moremessages. The method also includes initiating delivery of the messagecampaign to the plurality of users and receiving, from a plurality ofclient devices associated with the plurality of users, campaign-feedbackdata. The campaign-feedback data is indicative of user interaction,during the message campaign, that corresponds to presentation of atleast one of the features identified by the one or more annotations. Themethod further includes, after receiving the campaign-feedback data,receiving, from a requestor, a request for analysis of the features ofthe one or more messages during the message campaign and in response toreceiving the request for analysis of the features of the one or moremessages obtaining a campaign report that provides information on one ormore correlations between the features and user interaction with thefeatures during the message campaign and providing the campaign reportto the requestor.

In accordance with some embodiments, a method is performed at a serversystem with one or more processors and memory. The method includesestablishing separate storage areas for data from a plurality ofdata-collection server systems that include a first data-collectionserver system and a second data-collection server system. The methodalso includes receiving, from the first data-collection server system,first information that corresponds to feedback data from a plurality ofusers associated with the first data-collection server system, where thefirst information was obtained from a first set of users associated withthe first data-collection system and the first information excludesinformation that reveals the identity of individual users in the firstset of users. The method further includes receiving, from the seconddata-collection server system, second information that corresponds tofeedback data from a plurality of users associated with the seconddata-collection server system, where the second information was obtainedfrom a second set of users associated with the second data-collectionsystem and the second information excludes information that reveals theidentity of individual users in the second set of users. The method alsoincludes storing the first information in a first storage area for datafrom the first data-collection server system, storing the secondinformation in a second storage area for data from the seconddata-collection server system, and providing access to the firstinformation and the second information through one or more gatekeeperprocesses running on the data-management server system that controlaccess to data received from the data-collection systems.

In accordance with some embodiments, a computer system (e.g., a clientsystem or server system) includes one or more processors, memory, andone or more programs, the one or more programs are stored in the memoryand configured to be executed by the one or more processors and the oneor more programs include instructions for performing the operations ofany of the methods described above. In accordance with some embodiments,a non-transitory computer readable storage medium has stored thereininstructions which when executed by one or more processors, cause acomputer system (e.g., a client system or server system) to perform theoperations of the methods described above.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the disclosed embodiments, referenceshould be made to the Description of Embodiments below, in conjunctionwith the following drawings in which like reference numerals refer tocorresponding parts throughout the figures.

FIG. 1 is a block diagram illustrating a distributed client-serversystem, in accordance with some embodiments.

FIG. 2 is a block diagram illustrating a client system or an organizersystem, in accordance with some embodiments.

FIG. 3 is a block diagram illustrating a data-collection server system,in accordance with some embodiments.

FIG. 4 is a block diagram illustrating a remote storage system, inaccordance with some embodiments.

FIG. 5 is a block diagram illustrating a coordinating server system, inaccordance with some embodiments.

FIG. 6 includes a flow chart illustrating communications between aclient system, an organizer system, a data-collection server system, anda remote storage system collecting and processing data related to one ormore message campaigns, in accordance with some embodiments.

FIG. 7 includes a flow chart illustrating a method for correlatingmessage interaction data with prior versions of user profiles, inaccordance with some embodiments.

FIG. 8 includes a flow chart illustrating a method for storingcampaign-tracking data at a remote storage system, in accordance withsome embodiments.

FIGS. 9A-9B include a flow chart illustrating a method for analyzingmessage campaign feedback data that is stored in a remote storagesystem, in accordance with some embodiments.

FIG. 10 includes a flow chart illustrating a method for providinginformation about a user's interaction with multiple message campaigns,in accordance with some embodiments.

FIG. 11 includes a flow chart illustrating a method for determining acorrelation between human-identified features and user interaction withmessages in a message campaign, in accordance with some embodiments.

FIG. 12 includes a flow chart illustrating a method for controllingaccess to feedback data from multiple different data-collection serversystems, in accordance with some embodiments.

FIG. 13 includes an example of inserting annotations into HTML code tomark human-identified features in messages in a message campaign.

DESCRIPTION OF EMBODIMENTS

The present invention relates generally to the field of computer datamanagement, and in particular, to a system and method for efficientlymanaging message campaign data. FIG. 1 includes a block diagramillustrating an exemplary distributed Client-Server System 100 forefficiently managing message campaign data. System 100 includes aplurality of Client Systems 102 (e.g., Client System 102-1) that receivemessages as part of message campaigns initiated by one or moreData-Collection Server Systems 106 (e.g., Data-Collection System 106-1).Data-Collection Server Systems 106 collect feedback data that isindicative of user interactions with the message campaigns and storesome of the data in Remote Storage System 108 and provide informationabout the feedback data such as campaign reports to Organizer Systems103. In some implementations Organizer System 103 and Data-CollectionServer System 106 are implemented as an integrated computer system(e.g., Organizer System 103 operates as a console for Data-CollectionServer System 106). A Coordinating Server System 109 coordinatescommunication between the Remote Storage System 108, Data-CollectionServer Systems 106, Organizer Systems 103, and Client Systems 102. ACommunication Network 120 connects Client Systems 102, Organizer Systems103, Data-Collection Server Systems 106, and Remote Storage System 108and Coordinating Server System 109 to each other. Communication Network120 optionally includes the Internet, one or more local area networks(LANs), one or more wide area networks (WANs), other types of networks,or a combination of such networks.

Client Systems 102 optionally include one or more Client Applications110 and Client Data 112. Client System 102-N receives messages (e.g.,email messages) from Data-Collection Server System 106-1 and,optionally, other data-collection server systems. For example, ClientSystem 102-N receives emails including offers of products, admission toevents, and/or services. After receiving the messages, Client System102-N provides information to a feedback recipient (e.g.,Data-Collection Server System 106-N or Remote Storage System 108) as touser interaction with the messages (e.g., by providing an indication ofwhether the messages have been opened, responded to, forwarded to otherrecipients, acted upon by taking an action specified by such messages,acted upon by taking an action made available by such messages such asactivating a link or other activatable element in the messages, deleted,blocked, etc.). In some circumstances, a message server system (e.g., ashort message service center (SMSC) or email server) receives messagesfor Client System 102 and forwards the messages to Client System 102when Client System 102 becomes available (e.g., when the intendedrecipient logs into their email account or turns on their phone toreceive SMS messages). In some embodiments (e.g., where a message serversystem forwards messages to Client System 102), certain statusinformation (e.g., information that an intended recipient address doesnot exist, or the intended recipient's mailbox is full) is generated andsent to Data-Collection Server System 106 by the message server system(optionally, without communicating with Client System 102), while otherstatus messages (e.g., information that a particular message was openedor an activatable element in the message was activated) are generated inresponse to actions that occur at Client System 102.

Organizer Systems 103 are used by message campaign organizers to runmessage campaigns and review feedback data related to the performance ofmessage campaigns (e.g., feedback data provided by Data-CollectionServer Systems 106). In some embodiments, Organizer Systems 103 getfeedback data directly from Remote Storage System 108. In someembodiments, a respective Organizer System 103 communicates withCoordinating Server System 109 to set a performance level ofData-Collection Server System 106 and Remote Storage System 108 withrespect to message campaigns that originate from (e.g., are initiatedby) the respective Organizer System 103. As used herein, a messagecampaign includes a program of one or more rounds (or stages) ofmessages that are sent to a set of users for a marketing purpose such asproviding information about products and services or building a brandrelationship. In some embodiments this set of users is two or moreusers. In some embodiments, a marketing campaign includes one or morepredefined rounds of messages that are sent to a predefined set of users(e.g., a back to school campaign that has an objective to push aparticular product to particular kinds of people in a particular timeframe). In some embodiments, a marketing campaign includes an ongoinginteractive program that implements marketing business rules, such as anew customer welcome or onboarding program.

Data-Collection Server System 106-1 includes Data Intake Module 130,Remote Storage Module 132, Analytics Module 134, Marketing Module 135,User Profile Storage 136, and Local Data Storage 138. In someembodiments, Marketing Module 135 initiates message campaigns and othermarketing programs, Data Intake Module 130 receives feedback data fromClient Systems 102 that corresponds to the message campaigns transmittedto Client Systems 102. Data Intake Module 130 optionally receivesinformation that enables a user profile to be stored in User ProfileStorage 136 (e.g., explicit user-selected preference informationreceived from Client Systems 102, implicit preference information basedon interactions with one or more message campaigns by users at ClientSystems 102, and/or pre-existing user preference information provided toData Intake Module 130 by or at the direction of Organizer System 103).In some embodiments, feedback data from Client Systems 102 is stored inLocal Data Storage 138 and/or is transmitted to Remote Storage Module132 for storage at Remote Storage System 108.

In some embodiments, feedback data from Client Systems 102 is providedconcurrently (or simultaneously) to both Local Data Storage 138 and alsoto Remote Storage Module 132 for storage in Remote Storage System 108(e.g., Staging Area 142 in Remote Storage System 108). For example,feedback data from Client Systems 102 is continuously synchronizedbetween Local Data Storage 138 and Remote Storage System 108 as feedbackdata is received. In some embodiments, feedback data from Client Systems102 is provided first to Local Data Storage 138 and is, at a later time(e.g., in accordance with a predefined synchronization schedule)synchronized with Remote Storage System 108. For example, feedback datais initially stored in Local Data Storage 138 throughout the day, and ata time with low system activity (e.g., at midnight or 2 am), data thatwas stored in Local Data Storage 138 and/or User Profile Storage 136throughout the day is retrieved by Remote Storage Module 132 (optionallywith user profile information from User Profile Storage 136) and istransmitted to Remote Storage System 108 (e.g., Staging Area 142 inRemote Storage System 108), so that all of the feedback data and userprofile information stored at Data-Collection Server System 106-1 iseventually transmitted to Remote Storage System 108, so that whenfeedback data or user profile information is deleted from Local DataStorage 138 and/or User Profile Storage 136, that data is stillavailable at Remote Storage System 108 for use by Data-Collection ServerSystem 106-1 (e.g., Analytics Module 134).

In some embodiments, feedback data is stored in Local Data Storage 138for a limited amount of time (e.g., 7, 14, 21 days or some otherreasonable amount of time) to enable the feedback data from recentmessage campaigns to be analyzed locally (e.g., by Analytics Module 134)and is deleted after this limited amount of time to conserve storagespace at Data-Collection Server System 106-1. In some embodiments,Analytics Module 134 pre-aggregates the feedback data and stores thepre-aggregated feedback data in Local Data Storage 138, where thepre-aggregated data corresponds to feedback data that is older than thelimited amount of time. This pre-aggregated data enables AnalyticsModule 134 to run a predefined set of analytics on the pre-aggregateddata while conserving storage space Data-Collection Server System 106-1.

Remote Storage System 108 is a high-volume storage system (e.g., aserver farm). In some embodiments, Data-Collection Server System 106-1and Remote Storage System 108 are owned and operated by a same entity(e.g., person or company). In some embodiments, Remote Storage System108 is a third-party storage system that is owned and operated by adifferent entity from the entity that owns and operates Data-CollectionServer System 106-1 (e.g., the entity that owns and operates DataCollection Server System 106-1 rents capacity from the owner of RemoteStorage System 108). One example of a third-party storage system isAMAZON REDSHIFT, which is a data warehouse service from which datastorage capacity and associated compute resources can be leased on aflexible basis. An advantage to using a data warehouse service that isrun by a third-party is that storage space and processing resources canbe purchased on an as-needed basis, so that the storage space andprocessing resources (and corresponding cost) can expand and contract asneeded to accommodate changing demand for storage of data and analyticsworkload.

Remote Storage System 108 receives data (e.g., feedback data forcorresponding message campaigns) from Data Collection Systems 106 and/orClient Systems 102. In some embodiments, the data is received by StorageInterface 140 and, as such, Storage Interface 140 governs storage of thedata in Staging Area 142 prior to being stored in Permanent Storage 144.In some embodiments, one or more Data-Collection Server Systems 106,Client Systems 102, and Organizer Systems 103 are enabled to write datadirectly to Staging Area 142 which is shared between these differentdata sources (e.g., Staging Area 142 operates as a write-only filetransfer server). Data Flow Control 146 controls the flow of data storedin Staging Area 142 into Permanent Storage 144 (e.g., by directing datato segments of Permanent Storage 144 associated with particulardata-collection systems). In some embodiments, Permanent Storage 144 islogically (and/or physically) segregated by data source, so that datafrom different Data-Collection Server Systems 106, or differentpredetermined groups of Data-Collection Server Systems 106, is directedto different, separate, areas of Permanent Storage 144, so as topreserve the confidentiality of the data stored in Permanent Storage144. Data stored in Permanent Storage 144 is provided to Data-CollectionSystems 106 on request. In some embodiments Analytics Module 148analyzes and/or aggregates data stored in Permanent Storage 144 based onqueries submitted to Storage Interface 140 by a requestor (e.g., adata-collection server system or a client system), and the analyzedand/or aggregated data is transmitted to the requestor (e.g., as acampaign report that corresponds to an analysis of feedback data fromone or more message campaigns).

Coordinating Server System 109 coordinates operations at Remote StorageSystem 108, Data-Collection Server Systems 106 and, optionally, ClientSystems 102. In some embodiments, Data-Collection Server Systems 106and/or software running thereon is purchased or leased from theowner/operator of Coordinating Server System 109, and CoordinatingServer System 109 includes administrative tools for controlling and/ormodifying the operation of Data-Collection Server Systems 106. Forexample, Coordinating Server System 109 optionally monitors theoperational status of Data-Collection Server Systems 106, provideslicense keys to Data-Collection Server Systems 106 that enable differentmessage campaign capabilities of Data-Collection Server Systems 106(e.g., setting a maximum rate at which messages can be sent out, amaximum number of recipients of a message campaign, an amount of datathat can be stored at Remote Storage System 108, etc.), and/or tracksdata usage of Data-Collection Server Systems 106 for billing purposes.Coordinating Server System 109 also, optionally, includes administrativetools for controlling and/or modifying the operation of Remote StorageSystem 108. For example, Coordinating Server System 109 optionally hasread/write access to Staging Area 142 for debugging purposes, andoptionally provides instructions to Remote Storage System 108 as to theservice level to provide to different Data-Collection Server Systems 106(e.g., a minimum requested data read/write speed and/or an amount ofreserved storage space for data from a particular Data-Collection ServerSystem 106).

FIG. 2 is a block diagram illustrating a Client System 102 or OrganizerSystem 103 in accordance with some embodiments. Client System 102 orOrganizer System 103 optionally includes Housing 201 and typicallyincludes one or more processing units CPU(s) 202, one or more network orother Communication Interfaces 204, Memory 206, User Interface 205comprising a display device and a keyboard, mouse, touchpad, touchscreenor other input device, and one or more Communication Buses 208 forinterconnecting these components. Memory 206 typically includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and optionally includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 206 optionally includesone or more storage devices remotely located from CPU(s) 202. Memory206, or alternatively the non-volatile memory device(s) within Memory206, comprises a non-transitory computer readable storage medium. Insome embodiments, Memory 206 or alternatively the non-transitorycomputer readable storage medium stores the following programs, modulesand data structures, or a subset thereof:

-   -   Operating System 210 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   Network Communication Module (or instructions) 212, for        connecting Client System 102 to other computers (e.g.,        Data-Collection Server System 106, Remote Storage System 108,        and/or Coordinating Server System 109) via one or more Network        Interfaces 204 (wired or wireless) and one or more Communication        Networks 120 (FIG. 1), such as the Internet, other wide area        networks, local area networks, metropolitan area networks, and        so on;    -   Client Application(s) 110, for performing user-specified        operations at Client System 102 and accessing, storing, and        generating Client Data 112, including one or more of:        -   Messaging Application 220 (e.g., an email, SMS, social            networking or other application that is capable of sending            and/or receiving electronic messages), for sending and/or            receiving electronic messages;        -   Web Browsing Application 222, for accessing websites and for            rendering content stored in a web-browser interpretable            formats (e.g., HTML. JavaScript, etc.), including webpages            and web applications;        -   Message Campaign Application 224 for generating, modifying,            and adding annotations to messages for message campaigns and            for creating message campaign definitions including rules            for running message campaigns;        -   Analytics Application 226, for requesting campaign reports            for message campaigns that indicate message campaign            recipient interaction with messages in the message            campaigns; and        -   Remote Storage Interface 228, for storing and/or retrieving            information (e.g., campaign reports, campaign-tracking data,            etc.) from Remote Storage System 108; and    -   Client Data 112, optionally including messages received as part        of a message campaign and/or campaign reports, campaign-tracking        data, campaign-feedback data, and/or data for use in generating        a new message campaign.

In some embodiments, the system is an Organizer System 103 that stores aMessage Campaign Application 224, Analytics Application 226 and RemoteStorage Interface 228. In some embodiments, the system is a ClientSystem that stores a Messaging Application 220 and a Web BrowsingApplication 222 but not a Message Campaign Application 224, AnalyticsApplication 226 or Remote Storage Interface 228.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, Memory 206optionally stores a subset of the modules and data structures identifiedabove. Furthermore, Memory 206 optionally stores additional modules anddata structures not described above.

FIG. 3 is a block diagram illustrating a Data-Collection Server System106 in accordance with some embodiments. Data-Collection Server System106 includes one or more processing units CPU(s) 302, one or morenetwork or other Communication Interfaces 304, Memory 306, optionally aUser Interface 305 comprising a display device and a keyboard, mouse,touchpad, touchscreen or other input device, and one or moreCommunication Buses 308 for interconnecting these components. Memory 306typically includes high-speed random access memory, such as DRAM, SRAM,DDR RAM or other random access solid state memory devices; andoptionally includes non-volatile memory, such as one or more magneticdisk storage devices, optical disk storage devices, flash memorydevices, or other non-volatile solid state storage devices. Memory 306optionally includes one or more storage devices remotely located fromthe CPU(s) 302. Memory 306, or alternatively the non-volatile memorydevice(s) within Memory 306, comprises a non-transitory computerreadable storage medium. In some embodiments, Memory 306 oralternatively the non-transitory computer readable storage medium storesthe following programs, modules and data structures, or a subsetthereof:

-   -   Operating System 310 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   Network Communication Module (or instructions) 312, for        connecting Data-Collection Server System 106 to other computers        (e.g., Client System 102, Organizer System 103, Remote Storage        System 108, and/or Coordinating Server System 109) via one or        more Network Interfaces 304 (wired or wireless) and one or more        Communication Networks 120 (FIG. 1), such as the Internet, other        wide area networks, local area networks, metropolitan area        networks, and so on;    -   Data Intake Module 130, for receiving campaign-feedback data        that indicates user interaction with messages sent to the users        as part of one or more message campaigns (e.g., message        campaigns initiated by Marketing Module 135);    -   Remote Storage Module 132, for controlling the flow of data to        and from Remote Storage System 108;    -   Analytics Module 134, for analyzing campaign-feedback data and        campaign-tracking data that is indicative of user interactions        with messages sent out as part of a message campaign;    -   Marketing Module 135, for running message campaigns, including:        -   Message Campaign Control Module 320, for controlling            delivery of messages in a message campaign based on Message            Campaign Definitions 322 and for directing the storage of            campaign-tracking data that is indicative of user            interactions with messages in a message campaign; and        -   Message Campaign Definitions 322, including rules for            generating messages in a message campaign based on            parameters of the message campaign, data retrieved from            external sources (e.g., weather, time of day, upcoming            holidays), and feedback from messages sent out as part of a            message campaign;    -   User Profile Storage 136, including one or more profiles 330 for        one or more users, with a respective User Profile 330-1        optionally comprising:        -   Unique ID 332-1 that is used to associate sensitive            information such as personally identifiable information in            User Profile 330 with information corresponding to the user            that is received from Remote Storage System 108 (in some            embodiments this identifier is globally unique meaning that            the same person is uniquely assigned an identifier and this            identifier is used across a plurality of messaging campaigns            that in which the respective person is participating);        -   Name 334 of the user for use in personalizing messages in a            message campaign;        -   Contact Information 335 such as an email address, phone            number, social network identifier, or the like, for            addressing messages to the user as part of a message            campaign;        -   User Demographics 336 for the user (e.g., age, geographic            location, income level, gender, ethnicity, etc.) for            determining a grouping of the user for analysis purposes;            and/or        -   User Preferences/Settings/Status 338 for personalizing            messages sent to the user in a message campaign and/or            selecting messages to deliver to the user as part of message            campaigns;    -   Local Data Storage 138, optionally including one or more of Raw        Campaign-Tracking Data 340 (e.g., data that indicates user        interaction with messages in a message campaign that has not        been aggregated and can be used to run generate newly generated        analytics that are not predefined) and Pre-Aggregated        Campaign-Tracking Data 342 (e.g., data that indicates user        interaction with messages in a message campaign that has been        aggregated so that it can be used to run one or more predefined        analytics).

In some embodiments, Marketing Module 135 (or another module withinData-Collection Server System 106) performs one or more of the functionsof a message transfer agent “MTA” or other form of electronic/digitalmessage sending system that communicates electronic/digital messages toend users, such as the message transfer agents described in U.S. Pat.No. 7,698,462, filed Oct. 22, 2007; U.S. Pat. Pub. No. 2010/0250477,filed Mar. 31, 2009; U.S. patent application Ser. No. 13/451,458, filedApr. 19, 2012; and U.S. patent application Ser. No. 13/705,135, filedDec. 4, 2012, all of which are hereby incorporated by reference in theirentirety.

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (e.g., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, Memory 306optionally stores a subset of the modules and data structures identifiedabove. Furthermore, Memory 306 optionally stores additional modules anddata structures not described above.

Although FIG. 3 shows a “Data-Collection Server System” 106, FIG. 3 isintended more as functional description of the various features whichmay be present in a set of servers than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, items shown separately could be combined andsome items could be separated. For example, some items shown separatelyin FIG. 3 could be implemented on single servers and single items couldbe implemented by one or more servers. The actual number of servers usedto implement Data-Collection Server System 106 and how features areallocated among them will vary from one implementation to another, andoptionally depends in part on the amount of data traffic that the systemmust handle during peak usage periods as well as during average usageperiods.

FIG. 4 is a block diagram illustrating a Remote Storage System 108 inaccordance with some embodiments. Remote Storage System 108 includes oneor more processing units CPU(s) 402, one or more network or otherCommunications Interfaces 404, Memory 406, and one or more CommunicationBuses 408 for interconnecting these components. Memory 406 includeshigh-speed random access memory, such as DRAM, SRAM, DDR RAM or otherrandom access solid state memory devices; and optionally includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 406 optionally includesone or more storage devices remotely located from the CPU(s) 402. Memory406, or alternatively the non-volatile memory device(s) within Memory406, comprises a non-transitory computer readable storage medium. Insome embodiments, Memory 406 or alternatively the non-transitorycomputer readable storage medium stores the following programs, modulesand data structures, or a subset thereof:

-   -   Operating System 410 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   Network Communication Module (or instructions) 412, for        connecting Remote Storage System 108 to other computers (e.g.,        Client System 102, Organizer System 103, Data-Collection Server        System 106, and/or Coordinating Server System 109) via one or        more Network Interfaces 404 (wired or wireless) and one or more        Communication Networks 120 (FIG. 1), such as the Internet, other        wide area networks, local area networks, metropolitan area        networks, and so on;    -   Storage Interface 140, for providing access to data stored at        Remote Storage System 108 including:        -   Data Request Receiver Module 420, for receiving requests for            data from Client System 102, Organizer System 103,            Data-Collection Server System 106, and/or Coordinating            Server System 109;        -   Target Data Retrieval Module 422, for retrieving data that            matches predefined criteria (e.g., retrieving            campaign-tracking information for particular users based on            a request that includes unique identifiers for the users);        -   Aggregate Data Retrieval Module 424, for retrieving            aggregated data for responding to requests for campaign            reports (e.g., executing analytics on raw campaign-tracking            data stored in Permanent Storage 144 to provide an analysis            of patterns of user interaction with messages in message            campaigns);    -   Data Flow Control 146, for directing the transfer of data        between Staging Area 142 and Permanent Storage 144;    -   Analytics Module 148, for executing queries and running other        analytics on data retrieved from Permanent Storage 144, for use        in generating reports (e.g., campaign reports or        message-interaction reports) that provide quantitative        information about the data stored in Permanent Storage 144;    -   Staging Area 142, for receiving data provided by Data-Collection        Server System(s) 106, Client System(s) 102, Organizer System(s)        103, and/or Coordinating Server System 109 before it is        transferred to Permanent Storage 144 by Data Flow Control 146        and for storing data to be retrieved directly by Data-Collection        Server System(s) 106, Client System(s) 102, Organizer System(s)        103, and/or Coordinating Server System 109 after it is        transferred from Permanent Storage 144 by Data Flow Control 146;    -   Permanent Storage 144, storing:        -   Raw Campaign-Tracking Data 426 (e.g., data that indicates            user interaction with messages in a message campaign that            has not been aggregated and can be used to run newly            generated analytics that are not predefined); and        -   Archived User Profile Data 428, including data from which a            plurality of different versions of a user profile for one or            more users can be generated, for a respective user, Unique            User ID 430-1 is stored in conjunction with data from which            a plurality of different versions of the profile for the            user can be generated (e.g., Profile Version 1 Data 432-1,            Profile Version 2 Data 432-2, etc.).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, Memory 406optionally stores a subset of the modules and data structures identifiedabove. Furthermore, Memory 406 optionally stores additional modules anddata structures not described above.

Although FIG. 4 shows a “Remote Storage System” 108, FIG. 4 is intendedmore as functional description of the various features which may bepresent in a set of servers than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, items shown separately could be combined andsome items could be separated. For example, some items shown separatelyin FIG. 4 could be implemented on single servers and single items couldbe implemented by one or more servers. The actual number of servers usedto implement Remote Storage System 108 and how features are allocatedamong them will vary from one implementation to another, and optionallydepends in part on the amount of data traffic that the system musthandle during peak usage periods as well as during average usageperiods.

FIG. 5 is a block diagram illustrating a Coordinating Server System 109in accordance with some embodiments. Coordinating Server System 109includes one or more processing units CPU(s) 502, one or more network orother Communications Interfaces 504, Memory 506, and one or moreCommunication Buses 508 for interconnecting these components. Memory 506includes high-speed random access memory, such as DRAM, SRAM, DDR RAM orother random access solid state memory devices; and optionally includesnon-volatile memory, such as one or more magnetic disk storage devices,optical disk storage devices, flash memory devices, or othernon-volatile solid state storage devices. Memory 506 optionally includesone or more storage devices remotely located from the CPU(s) 502. Memory506, or alternatively the non-volatile memory device(s) within Memory506, comprises a non-transitory computer readable storage medium. Insome embodiments, Memory 506 or alternatively the non-transitorycomputer readable storage medium stores the following programs, modulesand data structures, or a subset thereof:

-   -   Operating System 510 that includes procedures for handling        various basic system services and for performing hardware        dependent tasks;    -   Network Communication Module (or instructions) 512, for        connecting Coordinating Server System 109 to other computers        (e.g., Client System 102, Organizer System 103, Data-Collection        Server System 106, and/or Remote Storage System 108) via one or        more Network Interfaces 504 (wired or wireless) and one or more        Communication Networks 120 (FIG. 1), such as the Internet, other        wide area networks, local area networks, metropolitan area        networks, and so on;    -   Campaign Services Module 520, for setting parameters within        which one or more Data-Collection Server System(s) 106 operate        (e.g., by providing license authentication that enables        different message campaign service levels by adjusting the size        or speed of message campaigns) and monitoring usage of        Data-Collection Server Systems 106;    -   Remote Storage Management Module 522, for managing data stored        at Remote Storage System 108, adjusting the amount and quality        of storage allocated to different Data-Collection Server Systems        106, and, optionally, monitoring usage of Data-Collection Server        Systems 106;    -   Data-Collection Server System Interface(s) 524, for        communicating with Data-Collection Server Systems 106 and        performing administrative functions associated with        Data-Collection Server Systems 106 (e.g., providing technical        support and providing software and firmware updates); and    -   Remote Storage System Interface(s) 526, for communicating with        Remote Storage System 108 and performing administrative        functions associated with Remote Storage System 108 (e.g.,        providing software and firmware updates and managing data stored        at Remote Storage System 108).

Each of the above identified elements may be stored in one or more ofthe previously mentioned memory devices, and corresponds to a set ofinstructions for performing a function described above. The aboveidentified modules or programs (i.e., sets of instructions) need not beimplemented as separate software programs, procedures or modules, andthus various subsets of these modules may be combined or otherwisere-arranged in various embodiments. In some embodiments, Memory 506optionally stores a subset of the modules and data structures identifiedabove. Furthermore, Memory 506 optionally stores additional modules anddata structures not described above.

Although FIG. 5 shows a “Coordinating Server System” 109, FIG. 5 isintended more as functional description of the various features whichmay be present in a set of servers than as a structural schematic of theembodiments described herein. In practice, and as recognized by those ofordinary skill in the art, items shown separately could be combined andsome items could be separated. For example, some items shown separatelyin FIG. 5 could be implemented on single servers and single items couldbe implemented by one or more servers. The actual number of servers usedto implement Remote Coordinating Server System 109 and how features areallocated among them will vary from one implementation to another, andoptionally depends in part on the amount of data traffic that the systemmust handle during peak usage periods as well as during average usageperiods.

FIG. 6 shows a flowchart representing communications between one or moreClient Systems 102, one or more Organizer Systems 103, one or moreData-Collection Server Systems 106, and a Remote Storage System 108 whenanalyzing user interaction with a message campaign. Message campaignrecipients are, in some circumstances, client devices associated withusers who have signed up to receive message campaigns (e.g., promotionalemails or other electronic messages that provide information to usersabout goods and/or services or other information). In some embodiments,a user explicitly creates (602) a user profile. In some embodiments, auser profile is generated based on information collected from a userwhile the user is interacting with an organizer of the message campaign(e.g., making a purchase, responding to a message in a marketingcampaign, etc.). In some embodiments, an entity that controlsData-Collection Server System 106 acquires user profiles from anexternal source other than Client Systems 102. For example, in someembodiments, the user profiles are acquired from public domain sources(e.g., on line directories), or private sources (e.g., the entity paysanother entity for a customer list, the entity exchanges a firstcustomer list with another entity for a second customer list). In someembodiments, when a new user profile is generated, the new user profileis stored (604) at a Data-Collection Server Systems 106. In someembodiments, data about user interactions with one or more messagecampaigns is provided to Data-Collection Server System 106, whichcreates a user profile based on the data.

In some embodiments, a user profile for a particular user alreadyexists. In this case, information generated at Client System 102represents an update to an existing user profile stored either atData-Collection Server System 106 or at Remote Storage System 108. Insome embodiments, when Client System 102 receives or detects (605)information that corresponds to a change in an existing user profile,that information is transmitted to Data-Collection Server System 106. Insome embodiments, the user profile stored at Data-Collection ServerSystem 106 is updated (606) and the old version of the profile istransmitted to the Remote Storage System 108, and Remote Storage System108 archives (608) the old version of the profile (or archivesinformation from which the old version of the profile can bereconstructed, such as a delta between the old version of the profileand the current or prior version of the profile). A more detaileddiscussion of using archived versions of user profiles by correlatingmessage interaction data with prior versions of user profiles isdiscussed in greater detail below with reference to method 700.

In some embodiments, Data-Collection Server System 106 initiates (610) amessage campaign (e.g., at the direction of a message campaign organizervia an Organizer System 103). Initiation of a message campaign resultsin a plurality of messages being sent to a plurality of Client Systems102. Temporally, in some embodiments a message campaign last less than 1day, less than a week, less than a month, less than a season, less thana year or some other specified amount of times. In some embodiments amessage campaign does not have a predetermined termination time butrather is driven by a sales objective, such that that campaignterminates when the sales objective is achieved. In some embodiments amessage campaign does not have a predetermined termination time and isterminated or refined while the message campaign is in process inaccordance with the judgment of a campaign administrator.

One or more Client Systems 102 receive (612) one or more messages aspart of the message campaign. Once a particular message in the messagecampaign has been received by a respective Client System 102, therespective Client System 102 detects (614) a user's interaction with therespective message that is part of the message campaign. For example,the interaction of the user includes one or more of: opening themessage, deleting the message, marking the message as spam,unsubscribing from future message campaigns, interacting with content ofthe message such as by following a hyperlink in the message, purchasinga product or service offered in the message campaign, or the like. Oneor more Client Systems 102 collect data about user's interaction withthe message and transmit the collected data to Data-Collection ServerSystem 106.

In some embodiments, Data-Collection Server System 106 receives (616)feedback from a plurality of Client Systems 102 that includes data aboutthe user interaction with one or more messages in the message campaign.In some embodiments, the received feedback data is stored (618) locallyat Data-Collection Server System 106 for use in analyzing userinteractions with message campaigns. In some embodiments, the receivedfeedback data is redacted (620) by removing information that is notnecessary to analyze user interactions with message campaigns and/or issensitive data (e.g., personally identifiable information such as names,addresses, payment credentials, billing information, etc.) to createredacted data. In some embodiments, the redacted data is then stored(622) at the Remote Storage System 108 for later retrieval. In someembodiments, after the redacted data has been stored at Remote StorageSystem 108, Data-Collection Server System 106 deletes (624) thecorresponding locally stored feedback data, so as to conserve space atData-Collection Server System 106. In some embodiments, the locallystored feedback data is deleted in accordance with a predefined schedule(e.g., all feedback data over 7 days old is deleted). In someembodiments, the Remote Storage System 108 also stores (626) redacteddata for other data collection server systems (e.g., multiple differentData-Collection Server Systems 106 all store feedback data at RemoteStorage System 108). A more detailed discussion of storingcampaign-tracking data at a remote storage system is discussed ingreater detail below with reference to method 800.

In some embodiments, Organizer System 103 requests (628) analytics(e.g., analysis of user interactions with message campaigns initiated byOrganizer System 103) from Data-Collection Server System 106. Therequest for analytics is transmitted to Data-Collection Server System106. In response to the request, Data-Collection Server System 106determines whether or not to respond to the request using data storedlocally on Data-Collection Server System 106. In accordance with adetermination to respond to the request using (630) locally stored data,Data-Collection Server System 106 generates (632) a report based onanalysis of locally stored data and, optionally, other data retrievedfrom other sources (e.g., user profile data, a time of day, time ofyear, weather, sales data, etc.). In some embodiments, the other dataincludes environmental reference data such as financial market data. Forexample, for a report that includes information indicating aclick-through rate of recipients of a message campaign who are followingany company in the S&P 500, Data-Collection Server System 106 woulddetermine stock preference data from user profiles and combine the stockpreference information with information from a financial institutionindicating which companies are in the S&P 500 and feedback data from oneor more message campaigns to generate the requested report.

After generating the report, Data-Collection Server System 106 thentransmits the report to Organizer System 103. Organizer System 103reviews (634) the report received from Data-Collection Server System 106and, optionally, uses the results of the report to create future messagecampaigns and/or modify ongoing message campaigns so that the messagecampaigns are more effective and helpful for recipients. In accordancewith a determination not to respond to the request using (636) locallystored data (e.g., because locally stored data is not available orbecause Data-Collection Server System 106 is configured to use remotelystored data to respond to the request), Data-Collection Server System106 requests feedback data from Remote Storage System 108.

In response to receiving a request for feedback data, Remote StorageSystem 108 retrieves (638) the requested feedback data and, optionally,analyzes (640) the feedback data. In some embodiments the feedback dataretrieved by Remote Storage System 108 is analyzed in conjunction withdata retrieved (639) from other sources (e.g., a time of day, time ofyear, weather, sales data, time zone, profile data etc.). In someembodiments, Remote Storage System 108 prepares a report with theresults of the analysis of the feedback data. In some embodiments,Remote Storage System 108 prepares a preliminary analysis (e.g., byaggregating data in a manner that enables Data-Collection Server System106 to generate a report) and provides the preliminary analysis toData-Collection Server System 106 for further processing. In somecircumstances, Remote Storage System 108 performs similar operations fora plurality of different Data-Collection Server Systems 106. A moredetailed discussion of controlling access to feedback data from multipledifferent data-collection server systems is discussed in greater detailbelow with reference to method 1200.

After requesting feedback data from Remote Storage System 108,Data-Collection Server System 106 obtains (642) a report based on theanalysis of the remote data (e.g., by processing the informationreceived from Remote Storage System 108 to generate a report or by usingthe information received from Remote Storage System 108 as the report).Data-Collection Server System 106 then transmits the report to OrganizerSystem 103. Organizer System 103 reviews (634) the report received fromData-Collection Server System 106 and, optionally, uses the results ofthe report to create future message campaigns and/or modify ongoingmessage campaigns so that the message campaigns are more effective andhelpful for recipients. In some embodiments, Remote Storage System 108receives a request from the organizer system 103 via Data-CollectionServer System 106, analyzes the requested data, and responds toOrganizer System 103 directly, without sending the resulting data and/orreport to Data-Collection Server System 106. A more detailed discussionof analyzing message campaign feedback data that is stored in a remotestorage system is discussed in greater detail below with reference tomethods 900, 1000, and 1100.

FIG. 7 includes a flowchart representing a method 700 for correlatingmessage interaction data with prior versions of user profiles, accordingto certain embodiments. In particular, method 700 describes in greaterdetail operations performed at a server system (e.g., Data-CollectionServer System 106-1 as shown in FIG. 1), as discussed above in FIG. 6.Method 700 is, optionally, governed by instructions that are stored in anon-transitory computer readable storage medium and that are executed byone or more processors of one or more servers (e.g., Data-CollectionServer System 106, FIG. 3). Each of the operations shown in FIG. 700optionally corresponds to instructions stored in a computer memory orcomputer readable storage medium (e.g., memory 306 of Server 106 in FIG.3). The computer readable storage medium optionally includes a magneticor optical disk storage device, solid state storage devices such asFlash memory, or other non-volatile memory device or devices. Thecomputer readable instructions stored on the computer readable storagemedium optionally include one or more of: source code, assembly languagecode, object code, or other instruction format that is interpreted byone or more processors. Some operations in method 700 are, optionally,combined and/or the order of some operations are, optionally, changed.

In some embodiments, a server system (e.g., Data-Collection ServerSystem 106) obtains (702) user profiles for a plurality of users forwhich message interaction data is stored. In some embodiments, the userprofiles are stored at the server system (e.g., stored user profile330-1 as shown in FIG. 3). In some embodiments, the server systemmaintains storage of the user profiles at a remote storage system (e.g.,Remote Storage System 108). In some embodiments, the users are usersthat receive message campaigns at the direction of the server system, arespective message campaign of the plurality of message campaignsincludes one or more messages addressed to the plurality of users, andthe message interaction data includes campaign-feedback data that isindicative of user interaction with the messages transmitted to theplurality of client devices as part of the plurality of messagecampaigns. For example, a message campaign includes a set of emails tocustomers of an airline advertising upcoming flight deals that are beingoffered by the airline, with hyperlinks to webpages that initiate aprocess for purchasing plane tickets using the flight deals.

In some embodiments, the server system receives (704) profile-updateinformation indicative of a change to a respective profile of arespective user of the plurality of users (e.g., a user explicitlychanges a field of a user profile, or a user performs an operation thatis tracked as part of the user profile, such as purchasing over $500 ofgoods or services). In response to receiving the profile-updateinformation (706), the server system updates (708) the respectiveprofile of the respective user in accordance with the profile-updateinformation (e.g., updating user profile 606 as shown in FIG. 6). Theserver system then initiates (710) storage of profile-archiveinformation that enables the server system to obtain a prior version ofthe respective profile (e.g., a version of the respective profile thatexisted before the respective profile was updated in accordance with theprofile-update information). In some embodiments, the server systemstores a prior version of the profile. In some embodiments, the serverssystem stores a delta between the prior version of the profile and theupdated version of the profile or other information that enablesreconstruction of the prior version of the profile. Storing informationfrom which prior versions of user profiles can be recreated enables theserver system to determine how changes in user profiles correspond tochanges in user behavior which can be very helpful in generating moreeffective and helpful message campaigns.

In some embodiments, a remote storage system (e.g., Remote StorageSystem 108) that is separate and distinct from the server system storesprofile-archive information that enables the server system to obtainmultiple prior versions of profiles for each of a plurality of differentusers. In some embodiments, the prior version of the respective profileincludes an indication of a time range for which the prior version ofthe respective profile was a current version of the respective profile(e.g., the profile-archive information indicates a date on which therespective profile was updated, so that the server system can determinethat the user's rewards status was X from August 2012 to August 2013).Storing prior versions of user profiles for a large number of users willtake up a large amount of storage space, thus it is advantageous tostore the information that enables the prior versions of the userprofiles to be reconstructed in a remote storage system that can be moreeasily scaled than a local storage system.

In some embodiments, a current version of the respective profile for therespective user is stored in a local storage system and updating therespective profile for the respective user includes updating the currentversion of the respective profile at the local storage system. In someembodiments, the profile-archive information excludes personallyidentifiable information (e.g., names, social security numbers, creditcard information, billing address, etc.) and/or other sensitive userdata. Thus, in some embodiments, a complete copy of the respectiveprofile is not stored at a remote storage system (e.g., Remote StorageSystem 108) so as to ensure that personally identifiable informationabout users is not inadvertently compromised by the remote storagesystem.

In some embodiments, after updating the respective profile of therespective user, the server system receives (712), from a requestor,(e.g., a user of the server system, or a user of a client device that isin communication with the server system) a request for analysis of themessage interaction data. In response to receiving the request foranalysis of the message interaction data (714) the server system obtains(716) a message-interaction report that correlates the messageinteraction data with a set of one or more prior versions of profiles ofone or more users that includes the prior version of the respectiveprofile of the respective user. In some embodiments, obtaining themessage-interaction report includes retrieving data from the remotestorage system and generating the message-interaction report by runningone or more queries against the retrieved data. In some embodiments,obtaining the message-interaction report includes transmitting one ormore queries to the remote storage system and receiving results of thequeries from the remote storage system.

In some embodiments, the message-interaction report includes informationcorrelating the one or more prior versions of profiles of a respectiveuser with historical data that corresponds to the respective user (e.g.,a respective user's reward status 6 months ago is correlated with theuser's engagement with message campaigns from six months ago). In someembodiments, the message-interaction report includes information derivedby application of the one or more prior versions of profiles of one ormore users and historical data that corresponds to the one or more usersagainst a pattern classification technique. In some embodiments, thepattern classification technique includes a support vector machine,neural network, or clustering (e.g., user reward status is clusteredbased on purchase size to determine relationship between reward statusand purchase size across the users). In some embodiments, the patternclassification technique is trained on all or a portion of the messageinteraction data at a time prior to receiving the profile-updateinformation.

In some embodiments, the message-interaction report includes informationderived by application of the one or more prior versions of profiles ofthe plurality of users and historical data that corresponds to theplurality of users against a pattern classification technique. In someembodiments, the pattern classification technique includes a supportvector machine, neural network, or clustering (e.g., user reward statusis clustered based on purchase size to determine relationship betweenreward status and purchase size across the users). In some embodiments,the pattern classification technique is trained on all or a portion ofthe message interaction data at a time prior to receiving theprofile-update information.

In some embodiments, the message-interaction report includes informationcorrelating changes in a user profile of a respective user over timewith changes in behavior of the respective user over time (e.g., themessage-interaction report identifies changes in the user's level ofengagement with message campaigns and correlates the changes in theuser's level of engagement with changes in the respective user's rewardstatus over the same time period). In some embodiments, themessage-interaction report includes information derived by applicationof changes in the profile of one or more users over time and changes inbehavior of one or more users over time against a pattern classificationtechnique. In some embodiments, the pattern classification techniqueincludes a support vector machine, neural network, or clustering. Insome embodiments, the pattern classification technique is trained on allor a portion of the message interaction data at a time prior toreceiving the profile-update information.

In some embodiments, the message-interaction report includes informationderived by application of changes in the profile of a plurality of usersover time and changes in behavior of the plurality of users over timeagainst a pattern classification technique. In some embodiments, thepattern classification technique includes a support vector machine,neural network, or clustering (e.g., user reward status is clusteredbased on purchase size to determine relationship between reward statusand purchase size across the users). In some embodiments, the patternclassification technique is trained on all or a portion of the messageinteraction data at a time prior to receiving the profile-updateinformation. In some embodiments, after the message-interaction reporthas been obtained (e.g., generated by the server system or retrievedfrom a remote storage system) the server system provides (718) themessage-interaction report to the requestor.

Exemplary pattern classification techniques are disclosed in Duda,Richard, Pattern Classification, 2nd edition, New York, John Wiley &Sons, Inc., 2000, which is hereby incorporated by reference. In someembodiments, the pattern classification techniques described aboveinclude one or more of a two-way contingency table, three-waycontingency table, generalized linear model, logistic regression,loglinear model for a contingency table, multi-category logit model, ormodel for a matched pair. See, for example, Agresti, Alan, AnIntroduction to Categorical Data Analysis, New York, John Wiley & Sons,Inc., 1996, which is hereby incorporated by reference.

It should be understood that the particular order in which theoperations in FIG. 7 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 800, 900, 1000, 1100 and/or 1200) are also applicable in ananalogous manner to method 7 described above with respect to FIG. 700.For example, the messages, message campaigns, campaign-feedback data,and campaign-tracking data described above with reference to method 700optionally have one or more of the characteristics of the messages,message campaigns, campaign-feedback data, and campaign-tracking datadescribed herein with reference to other methods described herein (e.g.,methods 800, 900, 1000, 1100 and/or 1200). For brevity, these detailsare not repeated here.

FIG. 8 includes a flowchart representing a method 800 for storingcampaign-tracking data at a remote storage system, according to certainembodiments. In particular, method 800 describes in greater detailoperations performed at a server system (e.g., Data-Collection ServerSystem 106-1 as shown in FIG. 1), as discussed above in FIG. 6. Method800 is, optionally, governed by instructions that are stored in anon-transitory computer readable storage medium and that are executed byone or more processors of one or more servers (e.g., Data-CollectionServer System 106, FIG. 3). Each of the operations shown in FIG. 8optionally corresponds to instructions stored in a computer memory orcomputer readable storage medium (e.g., memory 306 of Server 106 in FIG.3). The computer readable storage medium optionally includes a magneticor optical disk storage device, solid state storage devices such asFlash memory, or other non-volatile memory device or devices. Thecomputer readable instructions stored on the computer readable storagemedium optionally include one or more of: source code, assembly languagecode, object code, or other instruction format that is interpreted byone or more processors. Some operations in method 800 are, optionally,combined and/or the order of some operations are, optionally, changed.

In some embodiments, a server system (e.g., Data-Collection ServerSystem 106) initiates (802) delivery of a plurality of message campaignsto a plurality of users, (e.g., customers) where a respective messagecampaign of the plurality of message campaigns includes one or moremessages addressed to at least a subset of the plurality of users (e.g.,initiating a message campaign 610 as shown in FIG. 6). In someembodiments, the server system receives (804), from a plurality ofclient devices associated with the plurality of users, campaign-feedbackdata. The campaign-feedback data is indicative of user interaction withthe messages transmitted to the plurality of client devices as part ofthe plurality of message campaigns (e.g., receiving feedback regardinginteraction 616, as shown in FIG. 6). In some embodiments, a respectivemessage campaign of the plurality of message campaigns includes messagessent using a plurality of different message types (e.g., messages sentas part of the respective message campaign include two or more of SMSmessages, emails, social networking messages, etc.).

In some embodiments, after receiving the campaign-feedback data, theserver system stores at least a subset of the campaign-tracking data ina local storage system (e.g., storing feedback data locally 618, asshown in FIG. 6). In some embodiments the server system deletesrespective campaign-tracking data stored in the local storage system(e.g., deleting feedback data locally 624, as shown in FIG. 6) inaccordance with predetermined criteria (e.g., delete campaign-trackingdata that is older than 7, 14, 28, 45 days, or some reasonable timeperiod, or delete old campaign-tracking data when the amount ofcampaign-tracking data stored in the local storage system exceeds anamount of storage that is allotted in the local storage system forstoring campaign-tracking data). Deleting campaign-tracking data storedin the local storage system conserves storage space on the local storagesystem and reduces the need for a large amount of storage at the serversystem, thereby reducing the cost of the server system. Additionally,storing campaign-feedback data in the local storage system improvesperformance of the server system by making recent data available foranalysis by the server system (e.g., to determine which types of messagecampaign have been the most effective in the last several days) andadjust future message campaigns based on that data.

In some embodiments, the server system receives a request for a reportgenerated using respective campaign-tracking data. In response toreceiving the request, the server system determines if the respectivecampaign-tracking data is available in the local storage system (e.g.,Local Storage 138 in FIGS. 1, 3). In accordance with a determinationthat the respective campaign-tracking data is available (e.g., Yes 630as shown in FIG. 6) in the local storage system, the server systemretrieves the respective campaign tracking data from the local storagesystem and generates the report based at least in part on the respectivecampaign-tracking data from the local storage system. In accordance witha determination that the respective campaign-tracking data is notavailable (e.g., No 636 as shown in FIG. 6) in the local storage system,(e.g., because the respective campaign-tracking data has been deletedfrom the local storage system) the server system retrieves therespective campaign tracking data from the remote storage system andgenerates the report based at least in part on the respectivecampaign-tracking data from the remote storage system. Thus, in someembodiments, if the date range of the query is outside of a date rangefor which local data is stored in a form that can be used to respond tothe query (e.g., as raw data or as pre-aggregated data that isaggregated in a way that enables it to be used to respond to the query),the data is retrieved from the remote storage system. For example, aquery across multiple campaigns that includes data more than one weekold and for which data has not been pre-aggregated cannot be performedusing local data if raw local data (e.g., data from which a query acrossmultiple campaigns could be generated) is discarded after a week. Forexample if data has been aggregated by country (e.g., U.S., Canada,Mexico, etc.), the aggregated data cannot be used to determine thedifferences in interaction with a message campaign between messagecampaign recipients in different states within the same country.

In some embodiments, the campaign-tracking data includes at least asubset of the campaign-feedback data. In some embodiments, afterreceiving the campaign-feedback data the server system generates thecampaign-tracking data based on the campaign-feedback data (e.g.,generate report based on analysis of local data 632 as shown in FIG. 6).In some embodiments, the server system generates the campaign-trackingdata by performing one or more of: removing personally identifiableinformation (e.g., removing personally identifiable information fromdata 620, as shown in FIG. 6) from the campaign-feedback data (e.g.,removing real names, passwords, credit card information, residentialaddresses and the like), adding information stored at a local storagesystem that corresponds to the campaign-feedback data (e.g., adding userreward status or information about a user's previous responses tocampaigns, or the like), combining the campaign-feedback data with datareceived from other sources that are related to the one or morecampaigns (e.g., social networking feeds related to the one or morecampaigns, electronic messages such as SMS and MMS messages related tothe one or more campaigns), normalizing campaign-feedback data receivedfrom a plurality of different sources to a common format used for thecampaign-tracking data (e.g., changing the order of data, appendingunique identifiers based on a particular user from which thecampaign-feedback data was received, and the like), and removinginformation that is not useful for tracking user interaction with theone or more campaigns. (e.g., removing static information such as theversion number of client software or the like).

In some embodiments, after receiving the campaign-feedback data, theserver system conditionally transmits (806), when a transmissioncriterion is satisfied, to one or more remote storage systems,campaign-tracking data that is based on the campaign-feedback data. Insome embodiments, while conditionally transmitting campaign trackingdata, the server system determines (808) whether the campaign-feedbackdata satisfies the transmission criterion. Satisfaction of thetransmission criterion is determined by a set of campaign-archivingcriteria and the set of campaign-archiving criteria include, but are notnecessarily limited to, a criterion that is met when thecampaign-feedback data corresponds to a message campaign for whichcampaign-feedback data is being archived. Other possiblecampaign-archiving criteria include: whether campaign archiving isenabled, whether the received campaign-feedback data is the type of datathat has been selected for archiving. In some embodiments, the set ofcampaign-archiving criteria can include any number of requirements butonly require a certain number or percentage of them to be met beforearchiving campaign data. In some embodiments, certain criteria aremandatory and others are not-mandatory such that for the campaign datato be archived all of the mandatory criteria must be satisfied and acertain number of non-mandatory criteria (or a certain percentage) mustbe satisfied. Conditionally transmitting data to a remote storage systemenables the server system to control which data is sent to the remotestorage system, so as to conserve storage space at the remote storagesystem and/or so as to prevent sensitive information from beingtransmitted to the remote storage system.

In accordance with a determination that the campaign-feedback datasatisfies the transmission criterion, the server system transmits (810),to the one or more remote storage systems, the campaign-tracking datathat is based on the campaign-feedback data. In some embodiments, thecampaign-tracking data is the campaign-feedback data. In someembodiments, the campaign-tracking data includes an annotated version ofthe campaign-feedback data. In accordance with a determination that thecampaign-feedback data does not satisfy the transmission criterion, theserver system forgoes (812) transmission of the campaign-tracking datato the one or more remote storage systems.

In some embodiments, the conditionally transmitting performed by theserver system comprises the server system determining whether thecampaign-feedback data satisfies the transmission criterion, where thetransmission criterion is a data transmission criterion. In accordancewith a determination that the campaign-feedback data meets the datatransmission criteria, (e.g., that a sufficient quantity of data hasbeen collected or that the data has been collected for a sufficientamount of time) the server system transmits the campaign-tracking datato the remote storage system. In accordance with a determination thatthe campaign-feedback data does not satisfy the data transmissioncriteria, (e.g., that a sufficient quantity of data has not beencollected and/or that the data has not been collected for a sufficientamount of time) the server system forgoes transmission of thecampaign-tracking data to the remote storage system. (e.g., until asufficient quantity of data has been collected or the data has beencollected for a sufficient amount of time).

In some embodiments, the server system has a respective amount ofallotted storage at the remote storage system. The server systemmonitors the quantity of campaign-tracking data that is transmitted tothe one or more remote storage systems. In response to determining thatthe respective amount of allotted storage at the one or more remotestorage systems is not sufficient to store all of the campaign-trackingdata, the server system transmits, to the one or more remote storagesystems, a request to increase the allotment of storage for storing thecampaign-tracking data at the one or more remote storage systems. Insome embodiments, a coordinating server system (e.g., CoordinatingServer System 109) monitors the usage of the one or more remote storagesystem by the server system and automatically adjusts the allottedstorage at the one or more remote storage systems for the server system.

In some embodiments, the campaign feedback-data is associated withsensitive user data (e.g., personally identifiable information such as aname, billing address, payment credentials, email address, and the like)of a set of one or more users that is available to the server system. Insome embodiments, the campaign-tracking data includes information fromthe campaign-feedback data that is associated with the set of one ormore users and excludes the sensitive user data (e.g., when generatingthe campaign-tracking data from the campaign-feedback data, the serversystem removes sensitive user data from the campaign-feedback data). Insome embodiments, after transmitting the campaign-tracking data to theremote storage system, the server system receives a request to initiatea message campaign directed to users who meet campaign-targetingcriteria (e.g., for a “win back” message campaign, thecampaign-targeting criteria includes users who, at one point in time,had a high level of interaction with message campaigns but who have hada lower level of interaction with more recent message campaigns) andtransmits a request to the remote storage system for campaign-targetinginformation enabling identification of users who meet thecampaign-targeting criteria by the server system.

In some embodiments, in response to the request to initiate the messagecampaign, the server system receives the campaign-targeting information.The campaign-targeting information enables identification a plurality ofusers who meet the campaign-targeting criteria but does not includesensitive user data for the plurality of users (e.g., names and contactinformation for the plurality of users are not included in thecampaign-targeting information because this information is not stored atthe remote storage system). In some embodiments, after receiving thecampaign-targeting information, the server system initiates a messagecampaign that is targeted to the plurality of users based on thecampaign-targeting information and the sensitive user data. For example,the server system selects users to target with a marketing campaignbased on unique identifiers of users from the campaign-targetinginformation received from the remote storage system and customizes themessages in the message campaign by inserting names of users and usinguser email addresses that are not included in the campaign-targetinginformation but are, instead, retrieved from local storage of the serversystem based on the unique identifiers included in thecampaign-targeting information.

In some embodiments, a respective remote storage system (e.g., RemoteStorage System 108) of the one or more remote storage systems has aplurality of performance levels and while using the respective remotestorage system at a first performance level, the server systemdetermines that a different performance level for the respective remotestorage system is needed (e.g., determining that improved remote storagesystem performance is needed and increasing the performance level of therespective remote storage system or determining that the current levelof remote storage system performance is not needed and that a lowerlevel of performance would be acceptable and more cost-effective anddecreasing the performance level of the respective remote storagesystem). In some embodiments, in response to the determination that adifferent performance level for the respective remote storage system isneeded, the server system requests a change in performance level of therespective remote storage system from the first performance level to asecond, different performance level. In some embodiments, theperformance level of the respective remote storage system includesvalues for one or more of: an amount of storage space available at therespective remote storage system, a speed of read access to therespective remote storage system, a speed of write access to therespective remote storage system, and shared use of storage servers atthe remote storage system (e.g., whether the server system has exclusiveuse of a cluster of storage servers at the remote storage system orwhether usage of the cluster of storage servers is shared with one ormore third-parties). In some embodiments, the request to change theperformance level of the respective remote storage system is sent to theremote storage system. In some embodiments, the request to change theperformance level of the remote storage system is sent to a coordinatingserver that coordinates service of the remote storage system formultiple data-collection server systems and the request is forwardedfrom the coordinating server system to the remote storage system.

In some embodiments, the server system has a set of message campaigncapabilities, including one or more capabilities related to initiatingmessage campaigns and one or more capabilities related to collectingcampaign-feedback data. In some embodiments, the message campaigncapabilities include one or more of: a rate at which messages can besent as part of a message campaign, a number of users to which a messagecampaign can be directed, an amount of campaign-tracking orcampaign-feedback data that can be stored locally or at a remote storagesystem. In some embodiments, before operating in accordance with arespective message campaign capability of the set of message campaigncapability, the server system communicates with a coordinating serversystem (e.g., Coordinating Server System 109) to verify that the serversystem is authorized to operate in accordance with the respectivemessage campaign capability. In some embodiments, in accordance with adetermination that the server system is authorized to operate inaccordance with the respective message campaign capability, therespective message campaign capability is enabled at the server system,while in accordance with a determination that the server system is notauthorized to operate in accordance with the respective message campaigncapability, the respective message campaign capability is disabled atthe server system. In some embodiments, the coordinating server systemmanages message campaign capabilities for a plurality of differentserver systems that manage message campaigns and collect data related tothe message campaigns. In some embodiments, the owner/operator of thecoordinating server system sells or leases these server systems tocustomers for use in managing message campaigns and collecting datarelated to the message campaigns. In some embodiments, the server systemis configured so that it can be remotely disabled if its license toperform message campaign capabilities has expired (e.g., because amarketer has not renewed a license to use message campaign softwareassociated with the server system). Licensing server systems fordifferent message campaign capabilities, enables the owner/operator ofthe coordinating server system to provide variable pricing for theserver systems based on an expected amount of use of the server systems,which makes the server systems more affordable to message campaignorganizers who send out smaller message campaigns. Additionally,licensing server systems for different message campaign capabilitiesenables the capabilities of server systems to be changed withoutchanging the hardware of the server system. Thus, if a message campaignorganizer needs to add message campaign capabilities to the serversystem, the message campaign organizer can do so by requestingactivation of the additional message campaign capabilities by theowner/operator of the coordinating server system rather than replacingthe server system.

It should be understood that the particular order in which theoperations in FIG. 8 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 900, 1000, 1100 and/or 1200) are also applicable in ananalogous manner to method 800 described above with respect to FIG. 8.For example, the messages, message campaigns, campaign-feedback data,and campaign-tracking data described above with reference to method 800optionally have one or more of the characteristics of the messages,message campaigns, campaign-feedback data, and campaign-tracking datadescribed herein with reference to other methods described herein (e.g.,methods 700, 900, 1000, 1100 and/or 1200). For brevity, these detailsare not repeated here.

FIGS. 9A and 9B includes a flowchart representing a method 900 foranalyzing message campaign feedback data that is stored in a remotestorage system, according to certain embodiments. In particular, method900 describes in greater detail operations performed at a server system(e.g., Data-Collection Server System 106-1 as shown in FIG. 1), asdiscussed above in FIG. 6. Method 900 is, optionally, governed byinstructions that are stored in a non-transitory computer readablestorage medium and that are executed by one or more processors of one ormore servers (e.g., Data-Collection Server System 106, FIG. 3). Each ofthe operations shown in FIG. 900 optionally corresponds to instructionsstored in a computer memory or computer readable storage medium (e.g.,memory 306 of Server 106 in FIG. 3). The computer readable storagemedium optionally includes a magnetic or optical disk storage device,solid state storage devices such as Flash memory, or other non-volatilememory device or devices. The computer readable instructions stored onthe computer readable storage medium optionally include one or more of:source code, assembly language code, object code, or other instructionformat that is interpreted by one or more processors. Some operations inmethod 900 are, optionally, combined and/or the order of some operationsare, optionally, changed.

In some embodiments, a server system (e.g., Data-Collection ServerSystem 106) initiates (902) delivery of a plurality of message campaignsto a plurality of users, (e.g., customers) where a respective messagecampaign of the plurality of message campaigns includes one or moremessages addressed to at least a subset of the plurality of users. Insome embodiments, the respective message campaign includes messages sentusing a plurality of different message types (e.g., messages sent aspart of the respective message campaign include two or more of SMSmessages, emails, social networking messages, etc.).

In some embodiments, the server system obtains (904) a set of analysiscriteria. In some embodiments, the server system receives (906), from aplurality of client devices (e.g., Client Systems 102) associated withthe plurality of users, campaign-feedback data (e.g., receiving feedbackregarding interaction 616, as shown in FIG. 6). The campaign-feedbackdata is indicative of user interaction with the messages transmitted tothe plurality of client devices as part of the plurality of messagecampaigns and the campaign-feedback data includes information thatenables generation of campaign reports that provide quantitativeinformation about one or more of the plurality of message campaigns. Insome embodiments, while receiving the campaign-feedback data, the serversystem aggregates (908) at least a subset of the campaign-feedback datato generate aggregated campaign data in accordance with the set ofanalysis criteria (e.g., aggregating the campaign-feedback data so as toenable the list of queries to be performed using the aggregated campaigndata). In some embodiments, the server system stores (910) theaggregated campaign data in a local storage system (e.g., storingfeedback data locally 618, as shown in FIG. 6). For example, if theserver system has information indicating that a campaign organizer wantsto review the effectiveness of a message campaign broken down by age andgeographic location of the recipient, then the server system wouldaggregate the campaign-feedback data so that user engagement with themessage campaign between users of different ages and in differentgeographic locations can be compared by the campaign organizer using thelocally stored aggregated campaign data.

In some embodiments, after receiving the campaign-feedback data, theserver system transmits (912), to one or more remote storage systems,campaign-tracking data that is based on the campaign-feedback data. Insome embodiments, the campaign-tracking data includes fine grainedinformation that enables analysis of the campaign-tracking data based onarbitrary analysis criteria. In some embodiments, the server systemstores recent campaign-tracking data that is from before a respectivedate in a local storage system. The recent campaign-tracking data storedat the local storage system enables the server system to execute queriesthat have a date range that does not extend beyond the respective dateusing data in the local storage system without retrieving data from theremote storage system. In some embodiments, the server system stores rawcampaign-tracking data for a limited period of time such as for 7, 14,28, 45 days, until the amount of campaign-tracking data stored in thelocal storage system exceeds an amount of storage that is allotted inthe local storage system for storing campaign-tracking data. In someembodiments pre-aggregated campaign-tracking data is stored for a longerperiod of time, but this pre-aggregated campaign-tracking data is notconfigured to be used for responding to arbitrary queries. Thecampaign-tracking data transmitted to the remote storage system includesat least a portion of the recent campaign-tracking data and enables theserver system to execute queries that have a date range that extendsbeyond the respective date. In some embodiments, the remote storagesystem has campaign-tracking data that is from both before and afterthan the respective date and thus the campaign-tracking data at theremote storage system can be used to execute queries that have a daterange that extends before and after the respective date. Storingpre-aggregated data in local storage while storing raw data in a remotestorage system is advantageous in many situations, because it enablesthe server system to be respond quickly to a set of frequently usedqueries that can be executed using the pre-aggregated data whilemaintaining the flexibility of the fine grained data stored at theremote storage system, which can be used to run queries that cannot berun on the aggregated data.

In some embodiments, after transmitting the campaign-tracking data tothe one or more remote storage systems, the server system receives(914), from a requestor, (e.g., a user of Organizer System 103) a firstrequest for analysis of the campaign-feedback information. In someembodiments, the first request for analysis of the campaign-feedbackinformation includes analysis criteria that were not available to theserver system at the time that the campaign-tracking data was stored inthe remote storage system. For example, users of the server system hadnot determined which queries would be run against the campaign-trackingdata and thus very fine grained data is stored so that the ability torun custom queries on the campaign-tracking data is preserved.

In some embodiments, the first request for analysis of thecampaign-feedback information analytics includes a request to generate apivot table with one or more user-selected metrics (e.g., events to becounted) and one or more user-selected dimensions (e.g., ways in whichthe event counts are divided). In some embodiments, the one or moreuser-selected metrics include a metric selected from the set consistingof: a number of people targeted by a message campaign, a number ofinvalid addresses in a message campaign, a number of message deliveryfailures in a message campaign, a message delivery failure rate in amessage campaign, a number of messages opened in a message campaign, amessage open rate in a message campaign, a number of messages in whichcontent was clicked on in a message campaign, a message click rate in amessage campaign, a number of messages that were converted into sales ina message campaign, a conversion rate at which messages were convertedto sales in a message campaign, an amount of revenue generated by amessage campaign, an average order value of orders placed in response toa message campaign, a number of unsubscription requests received inresponse to a message campaign, a unsubscription rate for users whoreceived a message campaign, a number complaints received about amessage campaign, and a rate at which complaints were received in amessage campaign. In some embodiments, the one or more user-selecteddimensions include a dimension selected from the set consisting of: timeat which a message campaign was sent, time at which responses to amessage campaign were received, message campaign identifier for amessage campaign, message campaign version for a message campaign, emaildomain to which messages in a message campaign were sent, target usersto which a message campaign was sent, message content regions in amessage of a message campaign, and audience segment of a messagecampaign.

In some embodiments, in response to receiving the first request foranalysis of the campaign-feedback information (916), the server systemobtains (918) a campaign report generated using the campaign-trackingdata from the one or more remote storage systems (e.g., the serversystem obtains a report based on analysis of remote data 642 as shown inFIG. 6). In some embodiments, obtaining the campaign report includesretrieving at least a subset of the campaign-tracking data from theremote storage system and generating the campaign report by running oneor more queries against the retrieved campaign-tracking data. In someembodiments, obtaining the campaign report includes transmitting one ormore queries to the remote storage system and receiving results of thequeries from the remote storage system. In some embodiments, after themessage-interaction report has been obtained (e.g., generated by theserver system or retrieved from a remote storage system) the serversystem provides (920) the campaign report to the requestor. For example,when a message campaign organizer requests a pivot table showing messageopen rates for a set of message campaigns divided by the age of theusers, the server system provides an organizer system (e.g., OrganizerSystem 103) with information enabling the requested pivot table to bedisplayed to the message campaign organizer.

In some embodiments, after storing the aggregated campaign data, theserver system receives (922) a second request for analysis of thecampaign-feedback information (e.g., a request that has a date rangeoutside of raw campaign-feedback data that is stored in the localstorage system). In response to receiving the second request foranalysis of the campaign-feedback information and in accordance with adetermination that the second request can be satisfied using dataaggregated in accordance with the set of analysis criteria, the serversystem generates (926) a campaign report that is responsive to thesecond request using the aggregated campaign data (e.g., when the secondrequest includes a query from the list of queries that the data wasaggregated to accommodate, then the aggregated campaign data can be usedto execute the query). In accordance with a determination that thesecond request cannot be satisfied using data aggregated in accordancewith the set of analysis criteria, obtaining a campaign report generatedusing the campaign-tracking data from the one or more remote storagesystems (e.g., when the second request includes a query that is not onthe list of queries that the data was aggregated to accommodate, thenthe aggregated campaign data cannot be used to execute the query and theserver system reverts to using the raw data stored on the remote storagesystem to execute the query). Thus, in some embodiments, locally storeddata is used to respond to generate predefined analytics, but remotelystored data is used to generate analytics that are not predefinedanalytics and cannot be generated using the locally stored data.

In some embodiments, a respective remote storage system (e.g., RemoteStorage System 108) of the one or more remote storage systems has aplurality of performance levels and while using the respective remotestorage system at a first performance level, the server systemdetermines that a different performance level for the respective remotestorage system is needed. For example, the server system determines thatimproved remote storage system performance is needed and increases theperformance level of the respective remote storage system or the serversystem determines that the current level of remote storage systemperformance is not needed and that a lower level of performance would beacceptable and more cost-effective and the server system decreases theperformance level of the respective remote storage system. In someembodiments, in response to the determination that a differentperformance level for the respective remote storage system is needed,the server system requests a change in performance level of therespective remote storage system from the first performance level to asecond, different performance level. The performance level of therespective remote storage system includes values for one or more of: anamount of storage space available at the respective remote storagesystem, a speed of read access to the respective remote storage system,a speed of write access to the respective remote storage system, andshared use of storage servers at the remote storage system (e.g.,whether the server system has exclusive use of a cluster of storageservers at the remote storage system or whether usage of the cluster ofstorage servers is shared with one or more third-parties). In someembodiments, the request to change the performance level of therespective remote storage system is sent to the remote storage system.In some embodiments, the request to change the performance level of theremote storage system is sent to a coordinating server (e.g.,Coordinating Server System 109) that coordinates service of the remotestorage system for multiple data-collection server systems and therequest is forwarded from the coordinating server system to the remotestorage system.

It should be understood that the particular order in which theoperations in FIG. 9 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 800, 1000, 1100 and/or 1200) are also applicable in ananalogous manner to method 900 described above with respect to FIG. 9.For example, the messages, message campaigns, campaign-feedback data,and campaign-tracking data described above with reference to method 900optionally have one or more of the characteristics of the messages,message campaigns, campaign-feedback data, and campaign-tracking datadescribed herein with reference to other methods described herein (e.g.,methods 700, 800, 1000, 1100 and/or 1200). For brevity, these detailsare not repeated here.

FIG. 10 includes a flowchart representing a method 1000 for providinginformation about a user's interaction with multiple message campaigns,according to certain embodiments. In particular, method 1000 describesin greater detail operations performed at a server system (e.g.,Data-Collection Server System 106-1 as shown in FIG. 1), as discussedabove in FIG. 6. Method 1000 is, optionally, governed by instructionsthat are stored in a non-transitory computer readable storage medium andthat are executed by one or more processors of one or more servers(e.g., Data-Collection Server System 106, FIG. 3). Each of theoperations shown in FIG. 10 optionally corresponds to instructionsstored in a computer memory or computer readable storage medium (e.g.,memory 306 of Server 106 in FIG. 3). The computer readable storagemedium optionally includes a magnetic or optical disk storage device,solid state storage devices such as Flash memory, or other non-volatilememory device or devices. The computer readable instructions stored onthe computer readable storage medium optionally include one or more of:source code, assembly language code, object code, or other instructionformat that is interpreted by one or more processors. Some operations inmethod 1000 are, optionally, combined and/or the order of someoperations are, optionally, changed.

In some embodiments, a server system (e.g., Data-Collection ServerSystem 106) initiates (1002) delivery of a plurality of messagecampaigns to a plurality of users that include a first campaign and asecond campaign, (e.g., customers) where the first campaign and thesecond campaign each include one or more messages addressed to at leasta subset of the plurality of users. The server system receives (1004),from a plurality of client devices (e.g., Client Systems 102) associatedwith the plurality of users, campaign-feedback data (e.g., receivingfeedback regarding interaction 616, as shown in FIG. 6) for the firstcampaign and the second campaign. The campaign-feedback data isindicative of user interaction with the messages transmitted to theplurality of client devices as part of the plurality of messagecampaigns. In some embodiments, a respective message campaign of theplurality of message campaigns includes messages sent using a pluralityof different message types (e.g., messages sent as part of therespective message campaign include two or more of SMS messages, emails,social networking messages, etc.).

In some embodiments, after receiving the campaign-feedback data, theserver system directs (1006) storage of campaign-tracking data thatcorresponds to the campaign-feedback data (e.g., storing thecampaign-tracking data at the server system and/or transmitting thecampaign-tracking data to one or more remote storage systems such asRemote Storage System 108). After directing storage of thecampaign-tracking data, the server system receives (1008), from arequestor (e.g., a message campaign organizer using Organizer System103), a request for a campaign report for a respective user of theplurality of users (e.g., request analytics 628, as shown in FIG. 6).

In some embodiments, in response to receiving the request for a campaignreport for the respective user (1010), the server system obtains (1012)a respective campaign report (e.g., the server system obtains a reportbased on analysis of remote data 642 as shown in FIG. 6) for therespective user, where the respective campaign report includesinformation indicative of the respective user's interactions with thefirst campaign and the respective user's interactions with the secondcampaign (e.g., the respective user's interactions with the firstcampaign and the second campaign are isolated from interactions by otherusers in the campaign report). Thus, in some embodiments, a campaignreport is generated for a single user or a campaign report is generatedfor multiple users, but breaks information about the users' interactionswith message campaigns so that the interactions of individual users withthe message campaigns can be determined.

In some embodiments, the campaign report for the respective userincludes information identifying the plurality of message campaigns thatwere sent to the respective user. In some embodiments, the campaignreport for the respective user includes information identifying a levelof user engagement with the plurality of message campaigns that weresent to the respective user (e.g., a message open rate for a pluralityof message campaigns sent to the user). In some embodiments, thecampaign report for the respective user includes information identifyingone or more actions taken by the user in response to receiving one ormore of the plurality of message campaigns (e.g., for each of aplurality of message campaign messages, providing an indication ofwhether a user opened the message, selected a hyperlink in the message,and/or purchased any goods or services described in the message). Insome embodiments, the campaign report includes a first record for thefirst message campaign that occurred at a first time and a second recordfor the second message campaign that occurred at a second time, thefirst record enables access to information about a first version of auser profile of the respective user at the first time, and the secondrecord enables access to information about a second version of the userprofile of the respective user at the second time that is different fromthe first version of the user profile. For example if the user had a“gold” reward status when the first message campaign was received andhad a “sliver” reward status when the second message campaign wasreceived, the campaign report for the user would indicate the rewardstatus as well as the user's interaction with the different messagecampaigns.

In some embodiments, the campaign report includes information for theplurality of users and the information is separated out by user toindicate the interactions of individual users with the first campaignand the second campaign. In some embodiments, the campaign reportincludes information indicating aggregate behavior of the respectiveuser over the plurality of message campaigns. For example, the campaignreport indicates a level of engagement of the user across a plurality ofcampaigns, or indicates a rate at which messages to the user areconverted into sales or other beneficial activity for the campaignorganizer across a plurality of campaigns. In some embodiments, thecampaign report includes one or more controls for selecting whichmessage campaigns are included in the report. For example, the campaignreport enables a user reviewing the campaign report to change a daterange of the report so that only message campaigns that occurred duringthe date range are included in the report or enables the user reviewingthe campaign report to select message campaigns to include in the reportbased on other user-defined criteria. Viewing a report that shows theresponse of a particular user to message campaigns, can provide valuableinformation about the user and how best to tailor message campaigns toengage the user's interest.

In some embodiments, the respective user is a member of a group of users(e.g., an group of users who have registered as members of the group oran implicit group of users who meet predefined demographic criteria suchas age, sex, income level, geographic location, etc.) and the campaignreport for the respective user includes a comparison between therespective user's interactions with the first campaign (and therespective user's interactions with the second campaign) andinteractions of the group of users with the first campaign. In someembodiments, after the message-interaction report has been obtained(e.g., generated by the server system or retrieved from a remote storagesystem) the server system provides (1014) the respective campaign reportto the requestor. Comparing the interactions of a particular user withmessage campaigns as compared to the group of similar users can providevaluable information for better tailoring future message campaigns tothe user as well as to other users in the group. While the examplesdescribed above refer to a campaign report that includes informationabout a first message campaign and a second message campaign for arespective user, it should be understood that the information in thecampaign report optionally includes information about the respectiveuser's interactions with any number of message campaigns. Optionally thecampaign report includes statistics and activity detail for therespective user over the respective user's entire history with thecompany/brand whose products/services are being promoted by the messagecampaigns (or some subset of that history such as a time-limited subsetof that history).

It should be understood that the particular order in which theoperations in FIG. 10 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 800, 900, 1100 and/or 1200) are also applicable in ananalogous manner to method 1000 described above with respect to FIG. 10.For example, the messages, message campaigns, campaign-feedback data,and campaign-tracking data described above with reference to method 1000optionally have one or more of the characteristics of the messages,message campaigns, campaign-feedback data, and campaign-tracking datadescribed herein with reference to other methods described herein (e.g.,methods 700, 800, 900, 1100 and/or 1200). For brevity, these details arenot repeated here.

FIG. 11 includes a flowchart representing a method 1100 for determininga correlation between human-identified features and user interactionwith messages in a message campaign, according to certain embodiments.In particular, method 1100 describes in greater detail operations at aserver system (e.g., Data-Collection Server System 106-1 as shown inFIG. 1), as discussed above in FIG. 6. Method 1000 is, optionally,governed by instructions that are stored in a non-transitory computerreadable storage medium and that are executed by one or more processorsof one or more servers (e.g., Data-Collection Server System 106, FIG.3). Each of the operations shown in FIG. 11 optionally corresponds toinstructions stored in a computer memory or computer readable storagemedium (e.g., memory 306 of Server 106 in FIG. 3). The computer readablestorage medium optionally includes a magnetic or optical disk storagedevice, solid state storage devices such as Flash memory, or othernon-volatile memory device or devices. The computer readableinstructions stored on the computer readable storage medium optionallyinclude one or more of: source code, assembly language code, objectcode, or other instruction format that is interpreted by one or moreprocessors. Some operations in method 1100 are, optionally, combinedand/or the order of some operations are, optionally, changed.

In some embodiments, a server system (e.g., Data-Collection ServerSystem 106) receives (1102), from a client device (e.g., OrganizerSystem 103), one or more message templates for use in generatingmessages for a message campaign that includes sending the one or moremessages to a plurality of users, where the one or more messagetemplates include a plurality of annotations indicating human-identifiedfeatures in the one or more messages of the message campaign (e.g.,features such as sections of content and/or images that were identifiedby a human marketer to aid in determining the effectiveness of themessage campaign). In some embodiments, the message campaign includesmessages sent using a plurality of different message types (e.g.,messages sent as part of the message campaign include two or more of SMSmessages, emails, social networking messages, etc.).

In some embodiments, a respective message of the message campaign isgenerated by populating a message template of the one or more messagetemplates with content that is customized for a recipient of therespective message. For example, a message template includes a pluralityof defined regions with associated annotations, as shown in FIG. 13where a region is defined by start tag ##SV_BEGIN_REGION## and an endtag ##SV_END_REGION##, and the server system inserts content into thedefined regions. In some embodiments, generating a message for a messagecampaign includes inserting a name of the recipient and/or selectingcontent that is likely to be of particular interest to the recipientinto an email that is customized for the recipient of the message.

In some embodiments, the plurality of annotations are incorporated intothe content of the one or more messages. In some embodiments, arespective annotation of the plurality of annotations includes a starttag and an end tag in a respective message, content of the respectivemessage is in between the start tag and the end tag, and the content ofthe respective message that is between the start tag and the end tag isassociated with the annotation. In some embodiments, the annotations areconfigured so that the content that is associated with annotations doesnot overlap (e.g., each portion of content of the respective message isassociated with at most a single annotation). In some embodiments,multiple annotations are associated with a same portion of content(e.g., to indicate that the content has a “happy” sentiment and that thecontent is a “value-oriented” advertisement). In some embodiments, theserver system, prior to initiating delivery of the message campaign tothe plurality of users, removes the plurality of annotations from theone or more messages. In some embodiments, the annotations reflect thejudgment of a human user (e.g., the annotations for a respective messageare machine-interpretable information that indicates a marketer'sinterpretation of the content of the respective message). For example, amessage campaign organizer reviews a draft message for a messagecampaign, or a message template for use in generating messages in anadvertising campaign and define regions of the draft message or messagetemplate based on the campaign organizer's subjective judgment as to thenature of different portions of the draft message or message template(e.g., whether an image conveys a “happy” sentiment, a “sad” sentiment,a “high energy” sentiment, or a “relaxed” sentiment, or whether an offerfor goods or services conveys a “low cost” message or a “high value”message).

In some embodiments, the server system initiates (1104) delivery of themessage campaign to the plurality of users (e.g., initiate messagecampaign 610, as shown in FIG. 6). In some embodiments, the messages inthe message campaign include the annotations. In some embodiments, themessages in the message campaign do not include the annotations butinclude the human-identified features that are associated with theannotations via annotation correlation data generated by the serversystem. In some embodiments, when a respective message is sent out aspart of a messaging campaign, annotation information identifyingfeatures of the messages as being associated with different annotationsis stored by the server system for later use in associating featureswith which a user has interacted with corresponding annotations.

In some embodiments, receives (1106), from a plurality of client devicesassociated with the plurality of users, campaign-feedback data. Thecampaign-feedback data is indicative of user interaction, during themessage campaign, that corresponds to presentation of at least one ofthe features identified by the one or more annotations (e.g., receivefeedback regarding interaction 616, as shown in FIG. 6). For example,when the server system receives information indicating that a user hasselected a hyperlink in a message, the server system determines whichfeature of the message included the hyperlink and which annotationcorresponds to the feature of the message that included the hyperlinkand associates the selection of the hyperlink with the annotation (e.g.,user X selected a hyperlink associated with a feature with a “happy”sentiment). After associating the selection of the hyperlink with theannotation, the server system stores or directs storage of feedback datathat indicates that the user interacted with the feature identified bythe annotation. Similar operations are performed for other interactionswith messages in the message campaign by other users, and the serversystem also stores or directs storage of campaign-feedback data thatcorresponds to these interactions to enable generation of campaignreports that correlate the human-identified features and interactionswith the features by recipients of the message campaigns.

In some embodiments, after receiving the campaign-feedback data, theserver system receives (1108), from a requestor, a request for analysisof the features of the one or more messages during the message campaign.In some embodiments, in response (1110) to receiving the request foranalysis of the features of the one or more messages, the server systemobtains (1112) a campaign report that provides information on one ormore correlations between the features and user interaction with thefeatures during the message campaign. In some embodiments, obtaining thecampaign report includes retrieving data from a remote storage systemand generating the campaign report by running one or more queriesagainst the retrieved data. In some embodiments, obtaining the campaignreport includes transmitting one or more queries to a remote storagesystem and receiving results of the queries from the remote storagesystem (e.g., the server system obtains a report based on analysis 642of remote data as shown in FIG. 6). In some embodiments, the campaignreport indicates which type of feature users are more likely to interactwith (e.g., determining that a feature with a “happy” image is morelikely to be selected by users than a similar feature with a “sad”image).

In some embodiments, during the message campaign a respective message inthe one or more messages with a plurality of features is sent to a setof users, the campaign-feedback data includes an indication ofinteractions by users in the set of users with the features in therespective message, and the campaign report provides informationcomparing user engagement between different features in the respectivemessage (e.g., the campaign report compares the effectiveness ofadvertisements identified as value-oriented advertising by theannotations as compared with advertisements identified asquality-oriented advertising by the annotations in the respectivemessage of the message campaign). For example, the campaign reportincludes information indicating whether 18-25 year old males are morelikely to interact with an advertisement for a respective product if isadvertised as being a good value or if it is advertised as being apremium product. This information can be provided to campaign organizersfor use in selecting whether to advertise similar products to 18-25 yearold males using value-oriented advertising (e.g., if 18-25 year oldmales are shown to interact more with the respective product advertisedas a good value), or quality-oriented advertising (e.g., if 18-25 yearold males are shown to interact more with the respective productadvertised as a premium product).

In some embodiments, during the message campaign a first message with aplurality of features in a first arrangement is sent to a first set ofusers and a second message with the plurality of features in a secondarrangement is sent to a second set of users. In some embodiments, thecampaign-feedback data includes an indication of interactions by usersin the first set of users with a respective feature in the firstarrangement and an indication of the interaction by users in the secondset of users with the respective feature in the second arrangement. Insome embodiments, the campaign report provides information comparinguser engagement between the respective feature in the first arrangementand the respective feature in the second arrangement (e.g., the campaignreport compares the effectiveness of the respective feature when it isdisplayed at a top of an email as compared with when it is displayed ina side bar of an email). For example, the campaign report includesinformation indicating whether a “happy” image at a top of a messageincreases the effectiveness of a message more or less than a “happy”image in a side bar of the message.

In some embodiments, during the message campaign a first message with aplurality of features in a first arrangement is sent to a first set ofusers and a second message with the plurality of features in a secondarrangement is sent to a second set of users. In some embodiments, thecampaign-feedback data includes an indication of interactions by usersin the first set of users with a first feature in a respective locationin the first arrangement and an indication of the interaction by usersin the second set of users with a second feature in the respectivelocation in the second arrangement. In some embodiments, and thecampaign report provides information comparing user engagement betweenthe first feature in the respective location and the second feature inthe respective location (e.g., the campaign report compares whether afirst advertisement or a second advertisement is more effective at a topof an email). For example, for a respective marketing campaign thatcould use either messages of format A or messages of format B, aninitial test batch of messages is sent out. The initial test batch ofmessages includes sending messages with format A to 5% of the targetusers and sending messages with format B to 5% of the target users.After receiving campaign-feedback data from the initial test batch ofmessages, the server system determines whether messages with format A orformat B are more effective, selects the more effective format and sendsthe rest of the messages in message campaign using the selected format.In some embodiments, after the message-interaction report has beenobtained (e.g., generated by the server system or retrieved from aremote storage system) the server system provides (1114) the campaignreport to the requestor. For example the campaign report is transmittedfrom the server system to an organizer system (e.g., Organizer System103) that is associated with an organizer of one or more messagecampaigns.

It should be understood that the particular order in which theoperations in FIG. 11 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 800, 900, 1000, and/or 1200) are also applicable in ananalogous manner to method 1100 described above with respect to FIG. 11.For example, the messages, message campaigns, campaign-feedback data,and campaign-tracking data described above with reference to method 1100optionally have one or more of the characteristics of the messages,message campaigns, campaign-feedback data, and campaign-tracking datadescribed herein with reference to other methods described herein (e.g.,methods 700, 800, 900, 1000, and/or 1200). For brevity, these detailsare not repeated here.

FIG. 12 includes a flowchart representing a method 1200 for controllingaccess to feedback data from multiple different data-collection serversystems, according to certain embodiments. In particular, method 1200describes in greater detail operations performed at a server system(e.g., Remote Storage System 108 as shown in FIG. 1), as discussed abovein FIG. 6. Method 1200 is, optionally, governed by instructions that arestored in a non-transitory computer readable storage medium and that areexecuted by one or more processors of one or more servers (e.g., RemoteStorage System 108, FIG. 4). Each of the operations shown in FIG. 12optionally corresponds to instructions stored in a computer memory orcomputer readable storage medium (e.g., memory 406 of Server 108 in FIG.4). The computer readable storage medium optionally includes a magneticor optical disk storage device, solid state storage devices such asFlash memory, or other non-volatile memory device or devices. Thecomputer readable instructions stored on the computer readable storagemedium optionally include one or more of: source code, assembly languagecode, object code, or other instruction format that is interpreted byone or more processors. Some operations in method 1200 are, optionally,combined and/or the order of some operations are, optionally, changed.

In some embodiments, a data-management server system (e.g., RemoteStorage System 108) establishes (1202) separate (e.g., physicallyseparate or logically separate) storage areas for data from a pluralityof data-collection server systems (e.g., Data-Collection Server Systems106) that include a first data-collection server system and a seconddata-collection server system (e.g., server systems that are operated bycompanies that send marketing messages to their customers). In someembodiments, the separate storage areas are at the data-managementserver system. In some embodiments, the separate storage areas are at aremote storage system. In some embodiments, the first data-collectionserver system collects campaign-feedback data related to interaction ofrecipients of a first set of one or more message campaigns with messagesof the first set of one or more message campaigns and the seconddata-collection server system collects campaign-feedback data related tointeraction of recipients of a second set of one or more messagecampaigns with messages of the second set of one or more messagecampaigns. In some embodiments, the first storage area and the secondstorage area are not directly accessible by the first data-collectionserver system and the second data-collection server system (e.g., thedata-collection server systems are not granted read/write access to atleast some of the data stored at the remote storage system, such as thedata stored in Permanent Storage 144).

In some embodiments, the data-management server system receives (1204),from the first data-collection server system, first information thatcorresponds to feedback data from a plurality of users associated withthe first data-collection server system. In some circumstances, thefirst information was obtained from a first set of users associated withthe first data-collection system and the first information excludesinformation that reveals the identity of individual users in the firstset of users. In some embodiments, the feedback data is associated withsensitive information at the first data-collection server system butthat sensitive information is not transmitted to the data-managementserver system (e.g., the campaign-tracking information is anonymized byremoving any personally identifiable information such as real name,social security number, credit card number, home address, etc.).

In some embodiments, the data-management server system receives (1206),from the second data-collection server system, second information thatcorresponds to feedback data from a plurality of users associated withthe second data-collection server system. In some circumstances, thesecond information was obtained from a second set of users associatedwith the second data-collection system and the second informationexcludes information that reveals the identity of individual users inthe second set of users. In some embodiments, the feedback data isassociated with sensitive information at the second data-collectionserver system but that sensitive information is not transmitted to thedata-management server system (e.g., the campaign-tracking informationis anonymized by removing any personally identifiable information suchas real name, social security number, credit card number, home address,etc.).

The data-management server system then stores (1208) the firstinformation in a first storage area for data from the firstdata-collection server system (e.g., a portion of Permanent Storage 144that is reserved for data from the first data-collection server system).In some embodiments, the first information is stored as a flat file inthe first storage area. The data-management server system also stores(1210) the second information in a second storage area for data from thesecond data-collection server system (e.g., a portion of PermanentStorage 144 that is reserved for data from the second data-collectionserver system). In some embodiments, the first storage area and thesecond storage area are not directly accessible by the firstdata-collection server system and the second data-collection serversystem (e.g., the data-collection server systems are not grantedread/write access to data stored in Permanent Storage 144 at the remotestorage system). In some embodiments, the first storage area and thesecond storage area are controlled by the data-management server system.In some embodiments, prior to storing the first information in the firststorage area and storing the second information in the second storagearea, the data-management server system stores the first information andthe second information in a common storage area (e.g., Staging Area142). Enabling data-collection server systems to read/write from StagingArea 142 improves the performance of Remote Storage System 108, becauseRemote Storage System 108 does not need to filter incoming data. In someembodiments, data is later moved to Permanent Storage 144, to whichData-Collection Server Systems 106 do not have direct access, and thusthe data in Permanent Storage 144 is protected from being inadvertentlyaccessed by a Data-Collection Server System 106 that should not haveaccess to the data. Moreover, removing sensitive information from thedata that is provided to Remote Storage System 108 is advantageous,because it ensures that even if the security of Remote Storage System108 is compromised, the sensitive information is not also compromised.

In some embodiments, the data-management server system provides (1212)access to the first information and the second information through oneor more gatekeeper processes running on the data-management serversystem that control access to data received from the data-collectionsystems. In some embodiments, providing access to the first informationand the second information includes receiving a request for respectiveinformation. In some embodiments, in response to the request thedata-management server system determines whether the request is anaggregate data request or a targeted data request. In accordance with adetermination that the request is a targeted data request, thedata-management server system uses a first gatekeeper process (e.g.,Target Data Retrieval Module 422) to locate the respective information,where the first gatekeeper process is configured to retrieve targeteddata using one or more database queries. In accordance with adetermination that the request is an aggregate data request, thedata-management server system uses a second gatekeeper process (e.g.,Aggregate Data Retrieval Module 424) that is different from the firstgatekeeper process to locate and aggregate the respective information,where the second gatekeeper process is configured to retrieve andaggregate data using one or more database queries (e.g., thedata-management server system uses a first gatekeeper processes forretrieving specific information such as data for a particular user anduses a second gatekeeper process for retrieving aggregated informationfor use in analyzing the feedback data). Using different gatekeeperprocesses for retrieving different types of information enables thedifferent gatekeeper processes to be optimized for the types of tasksthat they are likely to be asked to perform.

It should be understood that the particular order in which theoperations in FIG. 12 have been described is merely exemplary and is notintended to indicate that the described order is the only order in whichthe operations could be performed. One of ordinary skill in the artwould recognize various ways to reorder the operations described herein.Additionally, it should be noted that details of other processesdescribed herein with respect to other methods described herein (e.g.,methods 700, 800, 900, 1000, and/or 1100) are also applicable in ananalogous manner to method 1200 described above with respect to FIG. 12.For example, the messages, message campaigns, campaign-feedback data,and campaign-tracking data described above with reference to method 1200optionally have one or more of the characteristics of the messages,message campaigns, campaign-feedback data, and campaign-tracking datadescribed herein with reference to other methods described herein (e.g.,methods 700, 800, 900, 1000, and/or 1100). For brevity, these detailsare not repeated here.

Attention is now directed to FIG. 13, which includes an example ofinserting annotations into HTML code to mark human-identified featuresin messages in a message campaign. In FIG. 13, HTML code for a templatemessage for generating messages for a message campaign is shown. TheHTML code describes multiple tables for the template email, defining,for the tables: a table class, width, border width, cell spacing, cellpadding and background color. Inserted in the HTML code for table thatincludes one or more annotated features are annotations that define theextent of a region that includes the annotated features. In thisexample, a start tag “##SV_BEGIN_REGION[Slot1][˜offer1]##” indicates abeginning of the region and an end tag “##SV_END_REGION##” indicates theend of the region. In the example shown in FIG. 13, content insertedinto the one or more tables that are defined between the start tag andthe end tag is annotated as being in a region corresponding to slot 1,offer 1. The information about the annotations associated with differentfeatures is maintained (e.g., in the messages sent as part of themessage campaign or stored at the server system) so that userinteractions with the features can be associated with the annotationswhen analyzing user interactions with the features (e.g., when obtaininga campaign report that provides information on one or more correlationsbetween the features and user interaction with the features during themessage campaign).

The foregoing description, for purpose of explanation, has beendescribed with reference to specific embodiments. However, theillustrative discussions above are not intended to be exhaustive or tolimit the invention to the precise forms disclosed. Many modificationsand variations are possible in view of the above teachings. Theembodiments were chosen and described in order to best explain theprinciples of the invention and its practical applications, to therebyenable others skilled in the art to best utilize the invention andvarious embodiments with various modifications as are suited to theparticular use contemplated.

What is claimed is:
 1. A method, comprising: at a server systemcomprising one or more processors and memory: receiving, from a clientdevice, one or more message templates for use in generating messages fora message campaign that includes sending the one or more messages to aplurality of users, wherein the one or more message templates include aplurality of annotations indicating human-identified features in the oneor more messages of the message campaign; initiating delivery of themessage campaign to the plurality of users; receiving, from a pluralityof client devices associated with the plurality of users,campaign-feedback data, wherein, the campaign-feedback data isindicative of user interaction, during the message campaign, thatcorresponds to presentation of at least one of the features identifiedby the one or more annotations; and after receiving thecampaign-feedback data, receiving, from a requestor, a request foranalysis of the features of the one or more messages during the messagecampaign; and in response to receiving the request for analysis of thefeatures of the one or more messages: obtaining a campaign report thatprovides information on one or more correlations between the featuresand user interaction with the features during the message campaign; andproviding the campaign report to the requestor.
 2. The method of claim1, wherein the plurality of annotations are incorporated into thecontent of the one or more messages of the message campaign.
 3. Themethod of claim 2, wherein: a respective annotation of the plurality ofannotations includes a start tag and an end tag in a respective message;content of the respective message is in between the start tag and theend tag; and the content of the respective message that is between thestart tag and the end tag is associated with the annotation.
 4. Themethod of claim 2, including, prior to initiating delivery of themessage campaign to the plurality of users, removing the plurality ofannotations from the one or more messages.
 5. The method of claim 1,wherein, when a respective message is sent out as part of a messagingcampaign, annotation information identifying features of the messages asbeing associated with different annotations is stored for later use inassociating features with which a user has interacted with correspondingannotations.
 6. The method of claim 1, wherein: during the messagecampaign a respective message in the one or more messages with aplurality of features is sent to a set of users; the campaign-feedbackdata includes an indication of interactions by users in the set of userswith the features in the respective message; and the campaign reportprovides information comparing user engagement between differentfeatures in the respective message.
 7. The method of claim 1, wherein:during the message campaign a first message with a plurality of featuresin a first arrangement is sent to a first set of users and a secondmessage with the plurality of features in a second arrangement is sentto a second set of users; the campaign-feedback data includes anindication of interactions by users in the first set of users with arespective feature in the first arrangement and an indication of theinteraction by users in the second set of users with the respectivefeature in the second arrangement; and the campaign report providesinformation comparing user engagement between the respective feature inthe first arrangement and the respective feature in the secondarrangement.
 8. The method of claim 1, wherein: during the messagecampaign a first message with a plurality of features in a firstarrangement is sent to a first set of users and a second message withthe plurality of features in a second arrangement is sent to a secondset of users; the campaign-feedback data includes an indication ofinteractions by users in the first set of users with a first feature ina respective location in the first arrangement and an indication of theinteraction by users in the second set of users with a second feature inthe respective location in the second arrangement; and the campaignreport provides information comparing user engagement between the firstfeature in the respective location and the second feature in therespective location.
 9. The method of claim 1, wherein the annotationsreflect the judgment of a human user.
 10. The method of claim 1,wherein, the message campaign includes messages sent using a pluralityof different message types.
 11. The method of claim 1, wherein arespective message of the message campaign is generated by populating amessage template of the one or more message templates with content thatis customized for a recipient of the respective message.
 12. A serversystem, comprising: one or more processors; memory; and one or moreprograms, wherein the one or more programs are stored in the memory andconfigured to be executed by the one or more processors, the one or moreprograms including instructions for: receiving, from a client device,one or more message templates for use in generating messages for amessage campaign that includes sending the one or more messages to aplurality of users, wherein the one or more message templates include aplurality of annotations indicating human-identified features in the oneor more messages of the message campaign; initiating delivery of themessage campaign to the plurality of users; receiving, from a pluralityof client devices associated with the plurality of users,campaign-feedback data, wherein, the campaign-feedback data isindicative of user interaction, during the message campaign, thatcorresponds to presentation of at least one of the features identifiedby the one or more annotations; and after receiving thecampaign-feedback data, receiving, from a requestor, a request foranalysis of the features of the one or more messages during the messagecampaign; and in response to receiving the request for analysis of thefeatures of the one or more messages: obtaining a campaign report thatprovides information on one or more correlations between the featuresand user interaction with the features during the message campaign; andproviding the campaign report to the requestor.
 13. The server system ofclaim 12, wherein the plurality of annotations are incorporated into thecontent of the one or more messages.
 14. The server system of claim 13,wherein: a respective annotation of the plurality of annotationsincludes a start tag and an end tag in a respective message; content ofthe respective message is in between the start tag and the end tag; andthe content of the respective message that is between the start tag andthe end tag is associated with the annotation.
 15. The server system ofclaim 13, including, prior to initiating delivery of the messagecampaign to the plurality of users, removing the plurality ofannotations from the one or more messages.
 16. The server system ofclaim 12, wherein, when a respective message is sent out as part of amessaging campaign, annotation information identifying features of themessages as being associated with different annotations is stored forlater use in associating features with which a user has interacted withcorresponding annotations.
 17. The server system of claim 12, wherein:during the message campaign a respective message in the one or moremessages with a plurality of features is sent to a set of users; thecampaign-feedback data includes an indication of interactions by usersin the set of users with the features in the respective message; and thecampaign report provides information comparing user engagement betweendifferent features in the respective message.
 18. The server system ofclaim 12, wherein: during the message campaign a first message with aplurality of features in a first arrangement is sent to a first set ofusers and a second message with the plurality of features in a secondarrangement is sent to a second set of users; the campaign-feedback dataincludes an indication of interactions by users in the first set ofusers with a respective feature in the first arrangement and anindication of the interaction by users in the second set of users withthe respective feature in the second arrangement; and the campaignreport provides information comparing user engagement between therespective feature in the first arrangement and the respective featurein the second arrangement.
 19. The server system of claim 12, wherein:during the message campaign a first message with a plurality of featuresin a first arrangement is sent to a first set of users and a secondmessage with the plurality of features in a second arrangement is sentto a second set of users; the campaign-feedback data includes anindication of interactions by users in the first set of users with afirst feature in a respective location in the first arrangement and anindication of the interaction by users in the second set of users with asecond feature in the respective location in the second arrangement; andthe campaign report provides information comparing user engagementbetween the first feature in the respective location and the secondfeature in the respective location.
 20. A method, comprising: at acomputer system comprising one or more processors and memory:transmitting, to a server system, one or more message templates for usein generating messages for a message campaign that includes sending theone or more messages to a plurality of users, wherein the one or moremessage templates include a plurality of annotations indicatinghuman-identified features in the one or more messages of the messagecampaign; transmitting, to the server system, a request to initiatedelivery of the message campaign to the plurality of users; after themessage campaign has been initiated, transmitting, to the server system,a request for analysis of the features of the one or more messagesduring the message campaign; and receiving, from the server system, inresponse to the request for analysis of the features of the one or moremessages, a campaign report that provides information on one or morecorrelations between the features and user interaction with the featuresduring the message campaign.